Overview

Workflow Report

This is a HTML-ification of Workflow state (workflow.state) and the associated files in the workflow output directory. Below is the doc section of the workflow (.workflow file). 

Produce, transform and plot fio-output
======================================

This workflow assumes that the 'prep.workflow' has completed.
The comparison and plots consists of

  * bandwidth in KiB / seconds
  * latency is mean and in nano-seconds
  * IOPS are 'raw' / base-unit

The above in a bar-plot at iodepth=1, and a line-plot as a function of iodepth.

See Artifacts for the generated plots
The workflow started on 25-10-2022, 21:05:35, status on the steps is provided below.

If a workflow step (worklet / run) or test produced any artifacts, then they are listed here.

This is the configuration (.config file) with any variable-placeholders such as {{ local.env.HOME }} filled out. 

devices:
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme0n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme1n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme2n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme3n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme4n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme5n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme6n1
- driver_attachment: kernel
  labels:
  - dev
  - bdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/nvme7n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng0n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng1n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng2n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng3n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng4n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng5n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng6n1
- driver_attachment: kernel
  labels:
  - dev
  - cdev
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: /dev/ng7n1
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:41:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:42:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:43:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:44:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:45:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:46:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:47:00.0'
- driver_attachment: userspace
  labels:
  - dev
  - pcie
  - nvm
  - scc
  nsid: 1
  spdk_conf: null
  uri: '0000:48:00.0'
fio:
  bin: /opt/fio/bin/fio
  build:
    prefix: /opt/fio
  engines:
    io_uring:
      type: builtin
    io_uring_cmd:
      type: builtin
    libaio:
      type: builtin
    spdk_bdev:
      path: /root/git/spdk/build/fio/spdk_bdev
      type: external_preload
    spdk_nvme:
      path: /root/git/spdk/build/fio/spdk_nvme
      type: external_preload
    xnvme:
      path: /usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so
      type: external_dynamic
  repository:
    path: /root/git/fio
    remote: https://github.com/axboe/fio.git
    tag: fio-3.32
liburing:
  repository:
    path: /root/git/liburing
    remote: https://github.com/axboe/liburing.git
    tag: liburing-2.2
os:
  name: debian
  version: bullseye
spdk:
  build:
    prefix: /opt/spdk
  repository:
    branch: for_bdev_xnvme_v6
    path: /root/git/spdk
    remote: https://github.com/safl/spdk.git
transport:
  ssh:
    hostname: fabio
    port: 22
    username: root
xnvme:
  build:
    type: debug
  repository:
    path: /root/git/xnvme
    remote: https://github.com/OpenMPDK/xNVMe.git
    tag: v0.6.0

Steps

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.output'),
                                  'state': {'begin': 1666724735.1971772,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8816070556640625,
                                            'end': 1666724736.0787842,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_01.state')},
                       'cmd_02': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '96312.06 IOPS     376.22 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '96312.06 IOPS     376.22 MiB/s\n'
                                            '[2022-10-25 23:03:47.429048] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:03:47.429120] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:03:47.429132] bdevperf '
                                            '[2022-10-25 23:03:47.429142] '
                                            '--no-shconf [2022-10-25 '
                                            '23:03:47.429151] -l 0 [2022-10-25 '
                                            '23:03:47.429169] --huge-unlink '
                                            '[2022-10-25 23:03:47.429179] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:03:47.429188] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:03:47.429196] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:03:47.429205] --iova-mode=pa '
                                            '[2022-10-25 23:03:47.429219] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:03:47.429227] '
                                            '--match-allocations [2022-10-25 '
                                            '23:03:47.429235] '
                                            '--file-prefix=spdk_pid82863 '
                                            '[2022-10-25 23:03:47.429243] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:03:47.567686] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:03:47.609539] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:03:47.610229] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.output'),
                                  'state': {'begin': 1666724737.1061788,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.127448797225952,
                                            'end': 1666724748.2336276,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_02.state')},
                       'cmd_03': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.output'),
                                  'state': {'begin': 1666724748.236189,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.789337158203125,
                                            'end': 1666724749.025526,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_03.state')},
                       'cmd_04': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '91332.77 IOPS     356.77 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '91332.77 IOPS     356.77 MiB/s\n'
                                            '[2022-10-25 23:04:00.370178] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:04:00.370248] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:04:00.370259] bdevperf '
                                            '[2022-10-25 23:04:00.370268] '
                                            '--no-shconf [2022-10-25 '
                                            '23:04:00.370277] -l 0 [2022-10-25 '
                                            '23:04:00.370285] --huge-unlink '
                                            '[2022-10-25 23:04:00.370293] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:04:00.370302] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:04:00.370309] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:04:00.370317] --iova-mode=pa '
                                            '[2022-10-25 23:04:00.370325] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:04:00.370333] '
                                            '--match-allocations [2022-10-25 '
                                            '23:04:00.370341] '
                                            '--file-prefix=spdk_pid82888 '
                                            '[2022-10-25 23:04:00.370350] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:04:00.508790] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:04:00.557574] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:04:00.558236] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.output'),
                                  'state': {'begin': 1666724750.0590663,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.12565016746521,
                                            'end': 1666724761.1847165,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.output'),
                                  'state': {'begin': 1666724761.1874182,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7818958759307861,
                                            'end': 1666724761.969314,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_05.state')},
                       'cmd_06': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '91051.09 IOPS     355.67 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '91051.09 IOPS     355.67 MiB/s\n'
                                            '[2022-10-25 23:04:13.314791] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:04:13.314863] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:04:13.314874] bdevperf '
                                            '[2022-10-25 23:04:13.314882] '
                                            '--no-shconf [2022-10-25 '
                                            '23:04:13.314890] -l 0 [2022-10-25 '
                                            '23:04:13.314898] --huge-unlink '
                                            '[2022-10-25 23:04:13.314905] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:04:13.314913] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:04:13.314920] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:04:13.314927] --iova-mode=pa '
                                            '[2022-10-25 23:04:13.314935] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:04:13.314942] '
                                            '--match-allocations [2022-10-25 '
                                            '23:04:13.314949] '
                                            '--file-prefix=spdk_pid82913 '
                                            '[2022-10-25 23:04:13.314956] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:04:13.453309] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:04:13.493561] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:04:13.494240] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.output'),
                                  'state': {'begin': 1666724762.9952679,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11287784576416,
                                            'end': 1666724774.1081457,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_06.state')},
                       'cmd_07': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_uring_foo.conf': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.output'),
                                  'state': {'begin': 1666724774.1112564,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7770850658416748,
                                            'end': 1666724774.8883414,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_07.state')},
                       'cmd_08': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '87268.14 IOPS     340.89 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '87268.14 IOPS     340.89 MiB/s\n'
                                            '[2022-10-25 23:04:26.232081] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:04:26.232151] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:04:26.232162] bdevperf '
                                            '[2022-10-25 23:04:26.232171] '
                                            '--no-shconf [2022-10-25 '
                                            '23:04:26.232180] -l 0 [2022-10-25 '
                                            '23:04:26.232189] --huge-unlink '
                                            '[2022-10-25 23:04:26.232197] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:04:26.232205] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:04:26.232212] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:04:26.232221] --iova-mode=pa '
                                            '[2022-10-25 23:04:26.232229] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:04:26.232237] '
                                            '--match-allocations [2022-10-25 '
                                            '23:04:26.232245] '
                                            '--file-prefix=spdk_pid82940 '
                                            '[2022-10-25 23:04:26.232253] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:04:26.370496] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:04:26.425602] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:04:26.426289] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.output'),
                                  'state': {'begin': 1666724775.910652,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.134417533874512,
                                            'end': 1666724787.0450695,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_08.state')},
                       'cmd_09': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.output'),
                                  'state': {'begin': 1666724787.048074,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7892162799835205,
                                            'end': 1666724787.8372903,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_09.state')},
                       'cmd_10': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '87338.70 IOPS     341.17 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '87338.70 IOPS     341.17 MiB/s\n'
                                            '[2022-10-25 23:04:39.195818] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:04:39.195889] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:04:39.195900] bdevperf '
                                            '[2022-10-25 23:04:39.195909] '
                                            '--no-shconf [2022-10-25 '
                                            '23:04:39.195918] -l 0 [2022-10-25 '
                                            '23:04:39.195926] --huge-unlink '
                                            '[2022-10-25 23:04:39.195934] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:04:39.195942] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:04:39.195949] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:04:39.195957] --iova-mode=pa '
                                            '[2022-10-25 23:04:39.195966] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:04:39.195973] '
                                            '--match-allocations [2022-10-25 '
                                            '23:04:39.195981] '
                                            '--file-prefix=spdk_pid82965 '
                                            '[2022-10-25 23:04:39.195989] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:04:39.333951] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:04:39.381531] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:04:39.382192] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.output'),
                                  'state': {'begin': 1666724788.8770678,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.116129159927368,
                                            'end': 1666724799.993197,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_10.state')},
                       'cmd_100': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223720.83 IOPS     873.91 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223720.83 IOPS     873.91 MiB/s\n'
                                             '[2022-10-25 23:20:50.653460] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:20:50.653531] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:20:50.653542] bdevperf '
                                             '[2022-10-25 23:20:50.653551] '
                                             '--no-shconf [2022-10-25 '
                                             '23:20:50.653560] -l 0 '
                                             '[2022-10-25 23:20:50.653567] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:20:50.653575] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:20:50.653583] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:20:50.653591] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:20:50.653599] --iova-mode=pa '
                                             '[2022-10-25 23:20:50.653607] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:20:50.653614] '
                                             '--match-allocations [2022-10-25 '
                                             '23:20:50.653622] '
                                             '--file-prefix=spdk_pid84111 '
                                             '[2022-10-25 23:20:50.653629] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:20:50.792163] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:20:50.833572] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:20:50.834238] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.output'),
                                   'state': {'begin': 1666725760.3471627,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.106633186340332,
                                             'end': 1666725771.453796,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.output'),
                                   'state': {'begin': 1666725771.4565046,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8669767379760742,
                                             'end': 1666725772.3234813,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_101.state')},
                       'cmd_102': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223708.71 IOPS     873.86 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223708.71 IOPS     873.86 MiB/s\n'
                                             '[2022-10-25 23:21:03.647189] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:21:03.647258] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:21:03.647268] bdevperf '
                                             '[2022-10-25 23:21:03.647277] '
                                             '--no-shconf [2022-10-25 '
                                             '23:21:03.647286] -l 0 '
                                             '[2022-10-25 23:21:03.647295] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:21:03.647303] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:21:03.647311] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:21:03.647318] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:21:03.647326] --iova-mode=pa '
                                             '[2022-10-25 23:21:03.647334] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:21:03.647342] '
                                             '--match-allocations [2022-10-25 '
                                             '23:21:03.647350] '
                                             '--file-prefix=spdk_pid84137 '
                                             '[2022-10-25 23:21:03.647359] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:21:03.785565] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:21:03.829482] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:21:03.830158] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.output'),
                                   'state': {'begin': 1666725773.3505797,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_uring_foo.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.105631828308105,
                                             'end': 1666725784.4562116,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_102.state')},
                       'cmd_103': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.output'),
                                   'state': {'begin': 1666725784.4601681,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7901983261108398,
                                             'end': 1666725785.2503664,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_103.state')},
                       'cmd_104': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223714.32 IOPS     873.88 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223714.32 IOPS     873.88 MiB/s\n'
                                             '[2022-10-25 23:21:16.578379] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:21:16.578446] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:21:16.578457] bdevperf '
                                             '[2022-10-25 23:21:16.578465] '
                                             '--no-shconf [2022-10-25 '
                                             '23:21:16.578474] -l 0 '
                                             '[2022-10-25 23:21:16.578482] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:21:16.578489] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:21:16.578497] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:21:16.578504] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:21:16.578512] --iova-mode=pa '
                                             '[2022-10-25 23:21:16.578519] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:21:16.578526] '
                                             '--match-allocations [2022-10-25 '
                                             '23:21:16.578534] '
                                             '--file-prefix=spdk_pid84162 '
                                             '[2022-10-25 23:21:16.578541] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:21:16.717093] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:21:16.765505] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:21:16.766176] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.output'),
                                   'state': {'begin': 1666725786.274423,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.119341611862183,
                                             'end': 1666725797.3937645,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_104.state')},
                       'cmd_105': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.output'),
                                   'state': {'begin': 1666725797.3967102,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7889127731323242,
                                             'end': 1666725798.185623,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_105.state')},
                       'cmd_106': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223656.87 IOPS     873.66 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223656.87 IOPS     873.66 MiB/s\n'
                                             '[2022-10-25 23:21:29.521276] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:21:29.521350] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:21:29.521362] bdevperf '
                                             '[2022-10-25 23:21:29.521371] '
                                             '--no-shconf [2022-10-25 '
                                             '23:21:29.521380] -l 0 '
                                             '[2022-10-25 23:21:29.521388] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:21:29.521396] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:21:29.521404] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:21:29.521412] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:21:29.521420] --iova-mode=pa '
                                             '[2022-10-25 23:21:29.521428] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:21:29.521436] '
                                             '--match-allocations [2022-10-25 '
                                             '23:21:29.521443] '
                                             '--file-prefix=spdk_pid84187 '
                                             '[2022-10-25 23:21:29.521454] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:21:29.660283] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:21:29.701545] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:21:29.702222] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.output'),
                                   'state': {'begin': 1666725799.2229874,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.107008457183838,
                                             'end': 1666725810.3299959,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.output'),
                                   'state': {'begin': 1666725810.333197,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7849862575531006,
                                             'end': 1666725811.1181834,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_107.state')},
                       'cmd_108': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223746.27 IOPS     874.01 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223746.27 IOPS     874.01 MiB/s\n'
                                             '[2022-10-25 23:21:42.426968] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:21:42.427037] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:21:42.427047] bdevperf '
                                             '[2022-10-25 23:21:42.427056] '
                                             '--no-shconf [2022-10-25 '
                                             '23:21:42.427063] -l 0 '
                                             '[2022-10-25 23:21:42.427071] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:21:42.427078] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:21:42.427086] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:21:42.427093] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:21:42.427100] --iova-mode=pa '
                                             '[2022-10-25 23:21:42.427108] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:21:42.427125] '
                                             '--match-allocations [2022-10-25 '
                                             '23:21:42.427132] '
                                             '--file-prefix=spdk_pid84213 '
                                             '[2022-10-25 23:21:42.427139] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:21:42.565813] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:21:42.613477] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:21:42.614155] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.output'),
                                   'state': {'begin': 1666725812.1362538,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.110110998153687,
                                             'end': 1666725823.2463648,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_108.state')},
                       'cmd_109': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.output'),
                                   'state': {'begin': 1666725823.249396,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.781137228012085,
                                             'end': 1666725824.0305333,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.output'),
                                  'state': {'begin': 1666724799.9959278,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.79339599609375,
                                            'end': 1666724800.7893238,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_11.state')},
                       'cmd_110': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223741.17 IOPS     873.99 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223741.17 IOPS     873.99 MiB/s\n'
                                             '[2022-10-25 23:21:55.355943] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:21:55.356022] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:21:55.356033] bdevperf '
                                             '[2022-10-25 23:21:55.356042] '
                                             '--no-shconf [2022-10-25 '
                                             '23:21:55.356050] -l 0 '
                                             '[2022-10-25 23:21:55.356057] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:21:55.356065] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:21:55.356072] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:21:55.356079] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:21:55.356087] --iova-mode=pa '
                                             '[2022-10-25 23:21:55.356094] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:21:55.356102] '
                                             '--match-allocations [2022-10-25 '
                                             '23:21:55.356109] '
                                             '--file-prefix=spdk_pid84238 '
                                             '[2022-10-25 23:21:55.356116] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:21:55.494518] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:21:55.549627] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:21:55.550308] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.output'),
                                   'state': {'begin': 1666725825.0630252,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.172744035720825,
                                             'end': 1666725836.2357693,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_110.state')},
                       'cmd_111': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.output'),
                                   'state': {'begin': 1666725836.238611,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7857277393341064,
                                             'end': 1666725837.0243387,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_111.state')},
                       'cmd_112': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223785.38 IOPS     874.16 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223785.38 IOPS     874.16 MiB/s\n'
                                             '[2022-10-25 23:22:08.359371] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:22:08.359442] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:22:08.359452] bdevperf '
                                             '[2022-10-25 23:22:08.359461] '
                                             '--no-shconf [2022-10-25 '
                                             '23:22:08.359470] -l 0 '
                                             '[2022-10-25 23:22:08.359478] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:22:08.359486] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:22:08.359494] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:22:08.359501] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:22:08.359511] --iova-mode=pa '
                                             '[2022-10-25 23:22:08.359519] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:22:08.359527] '
                                             '--match-allocations [2022-10-25 '
                                             '23:22:08.359535] '
                                             '--file-prefix=spdk_pid84265 '
                                             '[2022-10-25 23:22:08.359542] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:22:08.497812] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:22:08.545510] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:22:08.546171] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.output'),
                                   'state': {'begin': 1666725838.058469,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.185937643051147,
                                             'end': 1666725849.2444067,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.output'),
                                   'state': {'begin': 1666725849.247445,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7806620597839355,
                                             'end': 1666725850.0281072,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_113.state')},
                       'cmd_114': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223763.55 IOPS     874.08 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223763.55 IOPS     874.08 MiB/s\n'
                                             '[2022-10-25 23:22:21.359973] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:22:21.360044] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:22:21.360054] bdevperf '
                                             '[2022-10-25 23:22:21.360063] '
                                             '--no-shconf [2022-10-25 '
                                             '23:22:21.360071] -l 0 '
                                             '[2022-10-25 23:22:21.360079] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:22:21.360087] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:22:21.360095] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:22:21.360103] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:22:21.360110] --iova-mode=pa '
                                             '[2022-10-25 23:22:21.360118] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:22:21.360126] '
                                             '--match-allocations [2022-10-25 '
                                             '23:22:21.360134] '
                                             '--file-prefix=spdk_pid84290 '
                                             '[2022-10-25 23:22:21.360141] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:22:21.498321] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:22:21.545611] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:22:21.546279] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.output'),
                                   'state': {'begin': 1666725851.0596964,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_aio_foo.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.172929048538208,
                                             'end': 1666725862.2326255,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_114.state')},
                       'cmd_115': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.output'),
                                   'state': {'begin': 1666725862.2352889,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7849881649017334,
                                             'end': 1666725863.020277,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_115.state')},
                       'cmd_116': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223690.52 IOPS     873.79 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223690.52 IOPS     873.79 MiB/s\n'
                                             '[2022-10-25 23:22:34.349231] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:22:34.349305] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:22:34.349316] bdevperf '
                                             '[2022-10-25 23:22:34.349325] '
                                             '--no-shconf [2022-10-25 '
                                             '23:22:34.349333] -l 0 '
                                             '[2022-10-25 23:22:34.349342] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:22:34.349350] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:22:34.349358] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:22:34.349365] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:22:34.349374] --iova-mode=pa '
                                             '[2022-10-25 23:22:34.349383] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:22:34.349391] '
                                             '--match-allocations [2022-10-25 '
                                             '23:22:34.349399] '
                                             '--file-prefix=spdk_pid84315 '
                                             '[2022-10-25 23:22:34.349407] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:22:34.487798] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:22:34.529488] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:22:34.530150] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.output'),
                                   'state': {'begin': 1666725864.0558655,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.188930749893188,
                                             'end': 1666725875.2447963,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_116.state')},
                       'cmd_117': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.output'),
                                   'state': {'begin': 1666725875.2476728,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.8118739128112793,
                                             'end': 1666725876.0595467,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_117.state')},
                       'cmd_118': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223783.71 IOPS     874.16 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223783.71 IOPS     874.16 MiB/s\n'
                                             '[2022-10-25 23:22:47.374691] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:22:47.374761] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:22:47.374772] bdevperf '
                                             '[2022-10-25 23:22:47.374781] '
                                             '--no-shconf [2022-10-25 '
                                             '23:22:47.374789] -l 0 '
                                             '[2022-10-25 23:22:47.374797] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:22:47.374806] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:22:47.374814] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:22:47.374821] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:22:47.374847] --iova-mode=pa '
                                             '[2022-10-25 23:22:47.374857] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:22:47.374866] '
                                             '--match-allocations [2022-10-25 '
                                             '23:22:47.374873] '
                                             '--file-prefix=spdk_pid84340 '
                                             '[2022-10-25 23:22:47.374887] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:22:47.513230] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:22:47.553617] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:22:47.554293] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.output'),
                                   'state': {'begin': 1666725877.0861018,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.163384199142456,
                                             'end': 1666725888.249486,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_118.state')},
                       'cmd_119': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.output'),
                                   'state': {'begin': 1666725888.2520986,
                                             'cmd': 'rm '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '|| true',
                                             'cwd': 'None',
                                             'elapsed': 0.7791037559509277,
                                             'end': 1666725889.0312023,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_119.state')},
                       'cmd_12': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '87084.15 IOPS     340.17 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '87084.15 IOPS     340.17 MiB/s\n'
                                            '[2022-10-25 23:04:52.128230] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:04:52.128295] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:04:52.128306] bdevperf '
                                            '[2022-10-25 23:04:52.128315] '
                                            '--no-shconf [2022-10-25 '
                                            '23:04:52.128323] -l 0 [2022-10-25 '
                                            '23:04:52.128331] --huge-unlink '
                                            '[2022-10-25 23:04:52.128339] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:04:52.128347] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:04:52.128355] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:04:52.128362] --iova-mode=pa '
                                            '[2022-10-25 23:04:52.128371] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:04:52.128379] '
                                            '--match-allocations [2022-10-25 '
                                            '23:04:52.128386] '
                                            '--file-prefix=spdk_pid82990 '
                                            '[2022-10-25 23:04:52.128394] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:04:52.266924] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:04:52.325526] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:04:52.326204] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.output'),
                                  'state': {'begin': 1666724801.8169408,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.112801551818848,
                                            'end': 1666724812.9297423,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_12.state')},
                       'cmd_120': {'output': 'Running I/O for 10 seconds...\n'
                                             '\n'
                                             ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                             '\t Nvme0n1             :  '
                                             '223745.85 IOPS     874.01 MiB/s\n'
                                             '\n'
                                             ' '
                                             '=============================================================\n'
                                             '\n'
                                             ' Total                       :  '
                                             '223745.85 IOPS     874.01 MiB/s\n'
                                             '[2022-10-25 23:23:00.358689] '
                                             'Starting SPDK v23.01-pre git '
                                             'sha1 fd41a2937 / DPDK 22.07.0 '
                                             'initialization...\n'
                                             '[2022-10-25 23:23:00.358758] [ '
                                             'DPDK EAL parameters: [2022-10-25 '
                                             '23:23:00.358768] bdevperf '
                                             '[2022-10-25 23:23:00.358777] '
                                             '--no-shconf [2022-10-25 '
                                             '23:23:00.358786] -l 0 '
                                             '[2022-10-25 23:23:00.358795] '
                                             '--huge-unlink [2022-10-25 '
                                             '23:23:00.358803] '
                                             '--log-level=lib.eal:6 '
                                             '[2022-10-25 23:23:00.358811] '
                                             '--log-level=lib.cryptodev:5 '
                                             '[2022-10-25 23:23:00.358819] '
                                             '--log-level=user1:6 [2022-10-25 '
                                             '23:23:00.358826] --iova-mode=pa '
                                             '[2022-10-25 23:23:00.358834] '
                                             '--base-virtaddr=0x200000000000 '
                                             '[2022-10-25 23:23:00.358842] '
                                             '--match-allocations [2022-10-25 '
                                             '23:23:00.358850] '
                                             '--file-prefix=spdk_pid84365 '
                                             '[2022-10-25 23:23:00.358858] ]\n'
                                             'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n'
                                             '[2022-10-25 23:23:00.497334] '
                                             'app.c: 586:spdk_app_start: '
                                             '*NOTICE*: Total cores available: '
                                             '1\n'
                                             '[2022-10-25 23:23:00.545520] '
                                             'reactor.c: 926:reactor_run: '
                                             '*NOTICE*: Reactor started on '
                                             'core 0\n'
                                             '[2022-10-25 23:23:00.546189] '
                                             'accel_sw.c: '
                                             '461:sw_accel_module_init: '
                                             '*NOTICE*: Accel framework '
                                             'software module initialized.\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.output'),
                                   'state': {'begin': 1666725890.067811,
                                             'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                    '--json '
                                                    '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                    '-q 8 -o 4096 -w randread '
                                                    "-t 10 -m '[0]'",
                                             'cwd': 'None',
                                             'elapsed': 11.190223217010498,
                                             'end': 1666725901.2580342,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_120.state')},
                       'cmd_13': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_xnvme_io_uring.conf': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.output'),
                                  'state': {'begin': 1666724812.9327962,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7896976470947266,
                                            'end': 1666724813.722494,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_13.state')},
                       'cmd_14': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '95399.03 IOPS     372.65 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '95399.03 IOPS     372.65 MiB/s\n'
                                            '[2022-10-25 23:05:05.060149] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:05:05.060219] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:05:05.060231] bdevperf '
                                            '[2022-10-25 23:05:05.060240] '
                                            '--no-shconf [2022-10-25 '
                                            '23:05:05.060249] -l 0 [2022-10-25 '
                                            '23:05:05.060257] --huge-unlink '
                                            '[2022-10-25 23:05:05.060265] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:05:05.060273] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:05:05.060280] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:05:05.060288] --iova-mode=pa '
                                            '[2022-10-25 23:05:05.060296] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:05:05.060304] '
                                            '--match-allocations [2022-10-25 '
                                            '23:05:05.060312] '
                                            '--file-prefix=spdk_pid83016 '
                                            '[2022-10-25 23:05:05.060321] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:05:05.198531] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:05:05.245527] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:05:05.246196] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.output'),
                                  'state': {'begin': 1666724814.740206,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.111697912216187,
                                            'end': 1666724825.851904,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_14.state')},
                       'cmd_15': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.output'),
                                  'state': {'begin': 1666724825.8548682,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7805404663085938,
                                            'end': 1666724826.6354086,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_15.state')},
                       'cmd_16': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '95115.58 IOPS     371.55 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '95115.58 IOPS     371.55 MiB/s\n'
                                            '[2022-10-25 23:05:17.970774] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:05:17.970842] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:05:17.970852] bdevperf '
                                            '[2022-10-25 23:05:17.970861] '
                                            '--no-shconf [2022-10-25 '
                                            '23:05:17.970869] -l 0 [2022-10-25 '
                                            '23:05:17.970877] --huge-unlink '
                                            '[2022-10-25 23:05:17.970886] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:05:17.970894] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:05:17.970902] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:05:17.970910] --iova-mode=pa '
                                            '[2022-10-25 23:05:17.970918] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:05:17.970926] '
                                            '--match-allocations [2022-10-25 '
                                            '23:05:17.970934] '
                                            '--file-prefix=spdk_pid83041 '
                                            '[2022-10-25 23:05:17.970943] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:05:18.109398] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:05:18.153610] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:05:18.154319] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.output'),
                                  'state': {'begin': 1666724827.6594691,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.105032682418823,
                                            'end': 1666724838.7645018,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_16.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.output'),
                                  'state': {'begin': 1666724838.7677443,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7861323356628418,
                                            'end': 1666724839.5538766,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_17.state')},
                       'cmd_18': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '95206.38 IOPS     371.90 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '95206.38 IOPS     371.90 MiB/s\n'
                                            '[2022-10-25 23:05:30.898804] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:05:30.898875] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:05:30.898887] bdevperf '
                                            '[2022-10-25 23:05:30.898896] '
                                            '--no-shconf [2022-10-25 '
                                            '23:05:30.898905] -l 0 [2022-10-25 '
                                            '23:05:30.898912] --huge-unlink '
                                            '[2022-10-25 23:05:30.898921] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:05:30.898929] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:05:30.898937] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:05:30.898945] --iova-mode=pa '
                                            '[2022-10-25 23:05:30.898953] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:05:30.898960] '
                                            '--match-allocations [2022-10-25 '
                                            '23:05:30.898968] '
                                            '--file-prefix=spdk_pid83066 '
                                            '[2022-10-25 23:05:30.898976] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:05:31.037583] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:05:31.081613] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:05:31.082310] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.output'),
                                  'state': {'begin': 1666724840.5913122,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.099512338638306,
                                            'end': 1666724851.6908245,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_18.state')},
                       'cmd_19': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_aio_foo.conf': No "
                                            'such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.output'),
                                  'state': {'begin': 1666724851.6936128,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7899956703186035,
                                            'end': 1666724852.4836085,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_19.state')},
                       'cmd_20': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '94374.27 IOPS     368.65 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '94374.27 IOPS     368.65 MiB/s\n'
                                            '[2022-10-25 23:05:43.831772] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:05:43.831843] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:05:43.831854] bdevperf '
                                            '[2022-10-25 23:05:43.831863] '
                                            '--no-shconf [2022-10-25 '
                                            '23:05:43.831872] -l 0 [2022-10-25 '
                                            '23:05:43.831880] --huge-unlink '
                                            '[2022-10-25 23:05:43.831888] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:05:43.831897] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:05:43.831905] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:05:43.831913] --iova-mode=pa '
                                            '[2022-10-25 23:05:43.831921] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:05:43.831929] '
                                            '--match-allocations [2022-10-25 '
                                            '23:05:43.831937] '
                                            '--file-prefix=spdk_pid83091 '
                                            '[2022-10-25 23:05:43.831946] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:05:43.970469] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:05:44.009497] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:05:44.010185] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.output'),
                                  'state': {'begin': 1666724853.5132108,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.154639959335327,
                                            'end': 1666724864.6678507,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_20.state')},
                       'cmd_21': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.output'),
                                  'state': {'begin': 1666724864.6707578,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7889523506164551,
                                            'end': 1666724865.4597101,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_21.state')},
                       'cmd_22': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '94300.13 IOPS     368.36 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '94300.13 IOPS     368.36 MiB/s\n'
                                            '[2022-10-25 23:05:56.804496] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:05:56.804569] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:05:56.804581] bdevperf '
                                            '[2022-10-25 23:05:56.804590] '
                                            '--no-shconf [2022-10-25 '
                                            '23:05:56.804599] -l 0 [2022-10-25 '
                                            '23:05:56.804608] --huge-unlink '
                                            '[2022-10-25 23:05:56.804617] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:05:56.804625] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:05:56.804634] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:05:56.804643] --iova-mode=pa '
                                            '[2022-10-25 23:05:56.804652] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:05:56.804660] '
                                            '--match-allocations [2022-10-25 '
                                            '23:05:56.804670] '
                                            '--file-prefix=spdk_pid83116 '
                                            '[2022-10-25 23:05:56.804679] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:05:56.943818] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:05:56.989578] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:05:56.990266] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.output'),
                                  'state': {'begin': 1666724866.492729,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.173961639404297,
                                            'end': 1666724877.6666906,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_22.state')},
                       'cmd_23': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.output'),
                                  'state': {'begin': 1666724877.6687572,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7861847877502441,
                                            'end': 1666724878.454942,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_23.state')},
                       'cmd_24': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '94145.55 IOPS     367.76 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '94145.55 IOPS     367.76 MiB/s\n'
                                            '[2022-10-25 23:06:09.798113] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:06:09.798184] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:06:09.798194] bdevperf '
                                            '[2022-10-25 23:06:09.798203] '
                                            '--no-shconf [2022-10-25 '
                                            '23:06:09.798211] -l 0 [2022-10-25 '
                                            '23:06:09.798219] --huge-unlink '
                                            '[2022-10-25 23:06:09.798228] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:06:09.798235] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:06:09.798243] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:06:09.798251] --iova-mode=pa '
                                            '[2022-10-25 23:06:09.798259] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:06:09.798267] '
                                            '--match-allocations [2022-10-25 '
                                            '23:06:09.798274] '
                                            '--file-prefix=spdk_pid83141 '
                                            '[2022-10-25 23:06:09.798282] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:06:09.937122] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:06:09.989458] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:06:09.990130] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.output'),
                                  'state': {'begin': 1666724879.4857738,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.173627376556396,
                                            'end': 1666724890.6594012,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_24.state')},
                       'cmd_25': {'output': 'rm: cannot remove '
                                            "'/tmp/spdk_bdev_xnvme_libaio.conf': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.output'),
                                  'state': {'begin': 1666724890.6621559,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7881510257720947,
                                            'end': 1666724891.450307,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_25.state')},
                       'cmd_26': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '92010.24 IOPS     359.42 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '92010.24 IOPS     359.42 MiB/s\n'
                                            '[2022-10-25 23:06:22.801816] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:06:22.801884] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:06:22.801895] bdevperf '
                                            '[2022-10-25 23:06:22.801904] '
                                            '--no-shconf [2022-10-25 '
                                            '23:06:22.801913] -l 0 [2022-10-25 '
                                            '23:06:22.801920] --huge-unlink '
                                            '[2022-10-25 23:06:22.801928] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:06:22.801936] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:06:22.801943] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:06:22.801968] --iova-mode=pa '
                                            '[2022-10-25 23:06:22.801976] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:06:22.801983] '
                                            '--match-allocations [2022-10-25 '
                                            '23:06:22.801990] '
                                            '--file-prefix=spdk_pid83166 '
                                            '[2022-10-25 23:06:22.802006] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:06:22.940465] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:06:22.981534] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:06:22.982224] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.output'),
                                  'state': {'begin': 1666724892.4849267,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.1791250705719,
                                            'end': 1666724903.6640518,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_26.state')},
                       'cmd_27': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.output'),
                                  'state': {'begin': 1666724903.667292,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7869460582733154,
                                            'end': 1666724904.4542382,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_27.state')},
                       'cmd_28': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '92095.63 IOPS     359.75 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '92095.63 IOPS     359.75 MiB/s\n'
                                            '[2022-10-25 23:06:35.798932] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:06:35.799005] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:06:35.799016] bdevperf '
                                            '[2022-10-25 23:06:35.799025] '
                                            '--no-shconf [2022-10-25 '
                                            '23:06:35.799033] -l 0 [2022-10-25 '
                                            '23:06:35.799041] --huge-unlink '
                                            '[2022-10-25 23:06:35.799050] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:06:35.799058] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:06:35.799065] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:06:35.799075] --iova-mode=pa '
                                            '[2022-10-25 23:06:35.799082] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:06:35.799091] '
                                            '--match-allocations [2022-10-25 '
                                            '23:06:35.799099] '
                                            '--file-prefix=spdk_pid83193 '
                                            '[2022-10-25 23:06:35.799107] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:06:35.937685] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:06:35.993482] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:06:35.994154] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.output'),
                                  'state': {'begin': 1666724905.4874675,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.17349100112915,
                                            'end': 1666724916.6609585,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_28.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.output'),
                                  'state': {'begin': 1666724916.6637738,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7949099540710449,
                                            'end': 1666724917.4586837,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_29.state')},
                       'cmd_30': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :   '
                                            '91701.04 IOPS     358.21 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :   '
                                            '91701.04 IOPS     358.21 MiB/s\n'
                                            '[2022-10-25 23:06:48.806875] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:06:48.806943] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:06:48.806953] bdevperf '
                                            '[2022-10-25 23:06:48.806962] '
                                            '--no-shconf [2022-10-25 '
                                            '23:06:48.806970] -l 0 [2022-10-25 '
                                            '23:06:48.806977] --huge-unlink '
                                            '[2022-10-25 23:06:48.806985] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:06:48.806993] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:06:48.807000] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:06:48.807025] --iova-mode=pa '
                                            '[2022-10-25 23:06:48.807033] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:06:48.807041] '
                                            '--match-allocations [2022-10-25 '
                                            '23:06:48.807049] '
                                            '--file-prefix=spdk_pid83218 '
                                            '[2022-10-25 23:06:48.807064] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:06:48.945250] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:06:48.985598] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:06:48.986291] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.output'),
                                  'state': {'begin': 1666724918.4907243,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 1 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.182631492614746,
                                            'end': 1666724929.6733558,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_30.state')},
                       'cmd_31': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.output'),
                                  'state': {'begin': 1666724929.6761372,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7932009696960449,
                                            'end': 1666724930.4693382,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_31.state')},
                       'cmd_32': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '137677.93 IOPS     537.80 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '137677.93 IOPS     537.80 MiB/s\n'
                                            '[2022-10-25 23:07:01.817511] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:07:01.817582] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:07:01.817593] bdevperf '
                                            '[2022-10-25 23:07:01.817602] '
                                            '--no-shconf [2022-10-25 '
                                            '23:07:01.817611] -l 0 [2022-10-25 '
                                            '23:07:01.817618] --huge-unlink '
                                            '[2022-10-25 23:07:01.817628] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:07:01.817635] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:07:01.817643] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:07:01.817651] --iova-mode=pa '
                                            '[2022-10-25 23:07:01.817659] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:07:01.817667] '
                                            '--match-allocations [2022-10-25 '
                                            '23:07:01.817675] '
                                            '--file-prefix=spdk_pid83243 '
                                            '[2022-10-25 23:07:01.817684] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:07:01.956216] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:07:02.001505] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:07:02.002177] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.output'),
                                  'state': {'begin': 1666724931.5049977,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.115170955657959,
                                            'end': 1666724942.6201687,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_32.state')},
                       'cmd_33': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.output'),
                                  'state': {'begin': 1666724942.6232758,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7770218849182129,
                                            'end': 1666724943.4002976,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_33.state')},
                       'cmd_34': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '137638.53 IOPS     537.65 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '137638.53 IOPS     537.65 MiB/s\n'
                                            '[2022-10-25 23:07:14.742705] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:07:14.742773] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:07:14.742783] bdevperf '
                                            '[2022-10-25 23:07:14.742792] '
                                            '--no-shconf [2022-10-25 '
                                            '23:07:14.742800] -l 0 [2022-10-25 '
                                            '23:07:14.742808] --huge-unlink '
                                            '[2022-10-25 23:07:14.742815] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:07:14.742822] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:07:14.742829] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:07:14.742837] --iova-mode=pa '
                                            '[2022-10-25 23:07:14.742844] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:07:14.742851] '
                                            '--match-allocations [2022-10-25 '
                                            '23:07:14.742858] '
                                            '--file-prefix=spdk_pid83268 '
                                            '[2022-10-25 23:07:14.742866] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:07:14.881095] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:07:14.929550] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:07:14.930229] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.output'),
                                  'state': {'begin': 1666724944.4258957,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.125784158706665,
                                            'end': 1666724955.5516798,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_34.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.output'),
                                  'state': {'begin': 1666724955.5545845,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7928042411804199,
                                            'end': 1666724956.3473887,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_35.state')},
                       'cmd_36': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '122234.74 IOPS     477.48 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '122234.74 IOPS     477.48 MiB/s\n'
                                            '[2022-10-25 23:07:27.731543] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:07:27.731614] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:07:27.731625] bdevperf '
                                            '[2022-10-25 23:07:27.731633] '
                                            '--no-shconf [2022-10-25 '
                                            '23:07:27.731642] -l 0 [2022-10-25 '
                                            '23:07:27.731650] --huge-unlink '
                                            '[2022-10-25 23:07:27.731659] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:07:27.731666] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:07:27.731673] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:07:27.731681] --iova-mode=pa '
                                            '[2022-10-25 23:07:27.731689] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:07:27.731697] '
                                            '--match-allocations [2022-10-25 '
                                            '23:07:27.731705] '
                                            '--file-prefix=spdk_pid83293 '
                                            '[2022-10-25 23:07:27.731713] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:07:27.870333] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:07:27.917497] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:07:27.918172] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.output'),
                                  'state': {'begin': 1666724957.4195433,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.131767988204956,
                                            'end': 1666724968.5513113,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_36.state')},
                       'cmd_37': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.output'),
                                  'state': {'begin': 1666724968.554589,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7912216186523438,
                                            'end': 1666724969.3458107,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_37.state')},
                       'cmd_38': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '123370.19 IOPS     481.91 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '123370.19 IOPS     481.91 MiB/s\n'
                                            '[2022-10-25 23:07:40.684960] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:07:40.685028] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:07:40.685039] bdevperf '
                                            '[2022-10-25 23:07:40.685048] '
                                            '--no-shconf [2022-10-25 '
                                            '23:07:40.685057] -l 0 [2022-10-25 '
                                            '23:07:40.685065] --huge-unlink '
                                            '[2022-10-25 23:07:40.685072] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:07:40.685081] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:07:40.685088] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:07:40.685096] --iova-mode=pa '
                                            '[2022-10-25 23:07:40.685103] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:07:40.685112] '
                                            '--match-allocations [2022-10-25 '
                                            '23:07:40.685120] '
                                            '--file-prefix=spdk_pid83318 '
                                            '[2022-10-25 23:07:40.685128] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:07:40.823519] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:07:40.865686] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:07:40.866354] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.output'),
                                  'state': {'begin': 1666724970.3716917,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.107799053192139,
                                            'end': 1666724981.4794908,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_38.state')},
                       'cmd_39': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.output'),
                                  'state': {'begin': 1666724981.4824424,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7807235717773438,
                                            'end': 1666724982.263166,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_39.state')},
                       'cmd_40': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '123229.34 IOPS     481.36 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '123229.34 IOPS     481.36 MiB/s\n'
                                            '[2022-10-25 23:07:53.604306] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:07:53.604381] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:07:53.604392] bdevperf '
                                            '[2022-10-25 23:07:53.604401] '
                                            '--no-shconf [2022-10-25 '
                                            '23:07:53.604410] -l 0 [2022-10-25 '
                                            '23:07:53.604418] --huge-unlink '
                                            '[2022-10-25 23:07:53.604427] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:07:53.604434] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:07:53.604442] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:07:53.604450] --iova-mode=pa '
                                            '[2022-10-25 23:07:53.604458] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:07:53.604466] '
                                            '--match-allocations [2022-10-25 '
                                            '23:07:53.604474] '
                                            '--file-prefix=spdk_pid83343 '
                                            '[2022-10-25 23:07:53.604482] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:07:53.742580] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:07:53.813506] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:07:53.814176] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.output'),
                                  'state': {'begin': 1666724983.29524,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.133604764938354,
                                            'end': 1666724994.4288447,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_40.state')},
                       'cmd_41': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.output'),
                                  'state': {'begin': 1666724994.4316053,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7900600433349609,
                                            'end': 1666724995.2216654,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_41.state')},
                       'cmd_42': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '123447.21 IOPS     482.22 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '123447.21 IOPS     482.22 MiB/s\n'
                                            '[2022-10-25 23:08:06.563163] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:08:06.563231] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:08:06.563241] bdevperf '
                                            '[2022-10-25 23:08:06.563250] '
                                            '--no-shconf [2022-10-25 '
                                            '23:08:06.563258] -l 0 [2022-10-25 '
                                            '23:08:06.563267] --huge-unlink '
                                            '[2022-10-25 23:08:06.563276] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:08:06.563283] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:08:06.563291] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:08:06.563298] --iova-mode=pa '
                                            '[2022-10-25 23:08:06.563307] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:08:06.563315] '
                                            '--match-allocations [2022-10-25 '
                                            '23:08:06.563323] '
                                            '--file-prefix=spdk_pid83368 '
                                            '[2022-10-25 23:08:06.563331] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:08:06.701480] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:08:06.761538] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:08:06.762219] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.output'),
                                  'state': {'begin': 1666724996.2476013,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.12124228477478,
                                            'end': 1666725007.3688436,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_42.state')},
                       'cmd_43': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.output'),
                                  'state': {'begin': 1666725007.3717415,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7850737571716309,
                                            'end': 1666725008.1568153,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_43.state')},
                       'cmd_44': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '132739.00 IOPS     518.51 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '132739.00 IOPS     518.51 MiB/s\n'
                                            '[2022-10-25 23:08:19.508671] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:08:19.508741] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:08:19.508752] bdevperf '
                                            '[2022-10-25 23:08:19.508761] '
                                            '--no-shconf [2022-10-25 '
                                            '23:08:19.508770] -l 0 [2022-10-25 '
                                            '23:08:19.508778] --huge-unlink '
                                            '[2022-10-25 23:08:19.508786] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:08:19.508793] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:08:19.508801] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:08:19.508808] --iova-mode=pa '
                                            '[2022-10-25 23:08:19.508816] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:08:19.508824] '
                                            '--match-allocations [2022-10-25 '
                                            '23:08:19.508832] '
                                            '--file-prefix=spdk_pid83393 '
                                            '[2022-10-25 23:08:19.508840] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:08:19.647435] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:08:19.689565] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:08:19.690233] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.output'),
                                  'state': {'begin': 1666725009.1890814,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.122868299484253,
                                            'end': 1666725020.3119497,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_44.state')},
                       'cmd_45': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.output'),
                                  'state': {'begin': 1666725020.3147836,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8012819290161133,
                                            'end': 1666725021.1160655,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_45.state')},
                       'cmd_46': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '132646.77 IOPS     518.15 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '132646.77 IOPS     518.15 MiB/s\n'
                                            '[2022-10-25 23:08:32.474289] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:08:32.474360] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:08:32.474370] bdevperf '
                                            '[2022-10-25 23:08:32.474379] '
                                            '--no-shconf [2022-10-25 '
                                            '23:08:32.474389] -l 0 [2022-10-25 '
                                            '23:08:32.474397] --huge-unlink '
                                            '[2022-10-25 23:08:32.474404] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:08:32.474412] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:08:32.474419] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:08:32.474426] --iova-mode=pa '
                                            '[2022-10-25 23:08:32.474434] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:08:32.474442] '
                                            '--match-allocations [2022-10-25 '
                                            '23:08:32.474451] '
                                            '--file-prefix=spdk_pid83419 '
                                            '[2022-10-25 23:08:32.474458] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:08:32.612935] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:08:32.661606] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:08:32.662286] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.output'),
                                  'state': {'begin': 1666725022.1546354,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.117152690887451,
                                            'end': 1666725033.2717881,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_46.state')},
                       'cmd_47': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.output'),
                                  'state': {'begin': 1666725033.2747984,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7865428924560547,
                                            'end': 1666725034.0613413,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_47.state')},
                       'cmd_48': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '132765.93 IOPS     518.62 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '132765.93 IOPS     518.62 MiB/s\n'
                                            '[2022-10-25 23:08:45.526319] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:08:45.526390] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:08:45.526400] bdevperf '
                                            '[2022-10-25 23:08:45.526409] '
                                            '--no-shconf [2022-10-25 '
                                            '23:08:45.526418] -l 0 [2022-10-25 '
                                            '23:08:45.526426] --huge-unlink '
                                            '[2022-10-25 23:08:45.526433] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:08:45.526441] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:08:45.526448] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:08:45.526456] --iova-mode=pa '
                                            '[2022-10-25 23:08:45.526464] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:08:45.526472] '
                                            '--match-allocations [2022-10-25 '
                                            '23:08:45.526479] '
                                            '--file-prefix=spdk_pid83445 '
                                            '[2022-10-25 23:08:45.526487] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:08:45.665426] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:08:45.713532] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:08:45.714200] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.output'),
                                  'state': {'begin': 1666725035.0860467,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.245204210281372,
                                            'end': 1666725046.331251,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_48.state')},
                       'cmd_49': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.output'),
                                  'state': {'begin': 1666725046.3341322,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7895450592041016,
                                            'end': 1666725047.1236773,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_49.state')},
                       'cmd_50': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '131267.29 IOPS     512.76 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '131267.29 IOPS     512.76 MiB/s\n'
                                            '[2022-10-25 23:08:58.466029] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:08:58.466097] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:08:58.466108] bdevperf '
                                            '[2022-10-25 23:08:58.466116] '
                                            '--no-shconf [2022-10-25 '
                                            '23:08:58.466125] -l 0 [2022-10-25 '
                                            '23:08:58.466133] --huge-unlink '
                                            '[2022-10-25 23:08:58.466142] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:08:58.466149] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:08:58.466157] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:08:58.466172] --iova-mode=pa '
                                            '[2022-10-25 23:08:58.466180] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:08:58.466189] '
                                            '--match-allocations [2022-10-25 '
                                            '23:08:58.466197] '
                                            '--file-prefix=spdk_pid83471 '
                                            '[2022-10-25 23:08:58.466205] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:08:58.604528] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:08:58.657517] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:08:58.658187] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.output'),
                                  'state': {'begin': 1666725048.158771,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.167914152145386,
                                            'end': 1666725059.3266852,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_50.state')},
                       'cmd_51': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.output'),
                                  'state': {'begin': 1666725059.3299115,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7943465709686279,
                                            'end': 1666725060.124258,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_51.state')},
                       'cmd_52': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '131511.69 IOPS     513.72 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '131511.69 IOPS     513.72 MiB/s\n'
                                            '[2022-10-25 23:09:11.464489] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:09:11.464557] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:09:11.464568] bdevperf '
                                            '[2022-10-25 23:09:11.464577] '
                                            '--no-shconf [2022-10-25 '
                                            '23:09:11.464586] -l 0 [2022-10-25 '
                                            '23:09:11.464594] --huge-unlink '
                                            '[2022-10-25 23:09:11.464602] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:09:11.464611] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:09:11.464618] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:09:11.464626] --iova-mode=pa '
                                            '[2022-10-25 23:09:11.464634] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:09:11.464642] '
                                            '--match-allocations [2022-10-25 '
                                            '23:09:11.464650] '
                                            '--file-prefix=spdk_pid83497 '
                                            '[2022-10-25 23:09:11.464658] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:09:11.603992] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:09:11.645517] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:09:11.646198] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.output'),
                                  'state': {'begin': 1666725061.1545174,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.164811372756958,
                                            'end': 1666725072.3193288,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_52.state')},
                       'cmd_53': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.output'),
                                  'state': {'begin': 1666725072.32215,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7881443500518799,
                                            'end': 1666725073.1102943,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_53.state')},
                       'cmd_54': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '131606.51 IOPS     514.09 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '131606.51 IOPS     514.09 MiB/s\n'
                                            '[2022-10-25 23:09:24.448368] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:09:24.448436] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:09:24.448447] bdevperf '
                                            '[2022-10-25 23:09:24.448456] '
                                            '--no-shconf [2022-10-25 '
                                            '23:09:24.448465] -l 0 [2022-10-25 '
                                            '23:09:24.448473] --huge-unlink '
                                            '[2022-10-25 23:09:24.448481] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:09:24.448489] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:09:24.448496] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:09:24.448503] --iova-mode=pa '
                                            '[2022-10-25 23:09:24.448511] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:09:24.448520] '
                                            '--match-allocations [2022-10-25 '
                                            '23:09:24.448527] '
                                            '--file-prefix=spdk_pid83522 '
                                            '[2022-10-25 23:09:24.448535] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:09:24.587595] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:09:24.633513] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:09:24.634174] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.output'),
                                  'state': {'begin': 1666725074.1374826,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.185035228729248,
                                            'end': 1666725085.3225179,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_54.state')},
                       'cmd_55': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.output'),
                                  'state': {'begin': 1666725085.3256602,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7825493812561035,
                                            'end': 1666725086.1082096,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_55.state')},
                       'cmd_56': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '128907.87 IOPS     503.55 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '128907.87 IOPS     503.55 MiB/s\n'
                                            '[2022-10-25 23:09:37.447829] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:09:37.447899] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:09:37.447909] bdevperf '
                                            '[2022-10-25 23:09:37.447918] '
                                            '--no-shconf [2022-10-25 '
                                            '23:09:37.447926] -l 0 [2022-10-25 '
                                            '23:09:37.447934] --huge-unlink '
                                            '[2022-10-25 23:09:37.447943] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:09:37.447951] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:09:37.447958] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:09:37.447965] --iova-mode=pa '
                                            '[2022-10-25 23:09:37.447973] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:09:37.447981] '
                                            '--match-allocations [2022-10-25 '
                                            '23:09:37.447988] '
                                            '--file-prefix=spdk_pid83547 '
                                            '[2022-10-25 23:09:37.447996] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:09:37.586338] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:09:37.633485] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:09:37.634187] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.output'),
                                  'state': {'begin': 1666725087.1419184,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.141113996505737,
                                            'end': 1666725098.2830324,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_56.state')},
                       'cmd_57': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.output'),
                                  'state': {'begin': 1666725098.2859998,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7840650081634521,
                                            'end': 1666725099.0700648,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_57.state')},
                       'cmd_58': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '129471.51 IOPS     505.75 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '129471.51 IOPS     505.75 MiB/s\n'
                                            '[2022-10-25 23:09:50.417714] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:09:50.417786] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:09:50.417797] bdevperf '
                                            '[2022-10-25 23:09:50.417806] '
                                            '--no-shconf [2022-10-25 '
                                            '23:09:50.417815] -l 0 [2022-10-25 '
                                            '23:09:50.417823] --huge-unlink '
                                            '[2022-10-25 23:09:50.417830] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:09:50.417838] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:09:50.417845] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:09:50.417855] --iova-mode=pa '
                                            '[2022-10-25 23:09:50.417864] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:09:50.417872] '
                                            '--match-allocations [2022-10-25 '
                                            '23:09:50.417880] '
                                            '--file-prefix=spdk_pid83574 '
                                            '[2022-10-25 23:09:50.417889] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:09:50.556324] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:09:50.597526] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:09:50.598215] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.output'),
                                  'state': {'begin': 1666725100.084945,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.198548555374146,
                                            'end': 1666725111.2834935,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.output'),
                                  'state': {'begin': 1666725111.2860267,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7853360176086426,
                                            'end': 1666725112.0713627,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_59.state')},
                       'cmd_60': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '134840.52 IOPS     526.72 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '134840.52 IOPS     526.72 MiB/s\n'
                                            '[2022-10-25 23:10:03.412061] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:10:03.412135] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:10:03.412146] bdevperf '
                                            '[2022-10-25 23:10:03.412155] '
                                            '--no-shconf [2022-10-25 '
                                            '23:10:03.412164] -l 0 [2022-10-25 '
                                            '23:10:03.412171] --huge-unlink '
                                            '[2022-10-25 23:10:03.412179] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:10:03.412187] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:10:03.412195] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:10:03.412204] --iova-mode=pa '
                                            '[2022-10-25 23:10:03.412212] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:10:03.412220] '
                                            '--match-allocations [2022-10-25 '
                                            '23:10:03.412227] '
                                            '--file-prefix=spdk_pid83599 '
                                            '[2022-10-25 23:10:03.412236] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:10:03.550916] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:10:03.597661] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:10:03.598365] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.output'),
                                  'state': {'begin': 1666725113.1110542,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 2 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.184890270233154,
                                            'end': 1666725124.2959445,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_60.state')},
                       'cmd_61': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.output'),
                                  'state': {'begin': 1666725124.2987037,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7947700023651123,
                                            'end': 1666725125.0934737,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_61.state')},
                       'cmd_62': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '220666.80 IOPS     861.98 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '220666.80 IOPS     861.98 MiB/s\n'
                                            '[2022-10-25 23:10:16.626846] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:10:16.626916] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:10:16.626927] bdevperf '
                                            '[2022-10-25 23:10:16.626935] '
                                            '--no-shconf [2022-10-25 '
                                            '23:10:16.626943] -l 0 [2022-10-25 '
                                            '23:10:16.626951] --huge-unlink '
                                            '[2022-10-25 23:10:16.626958] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:10:16.626966] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:10:16.626973] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:10:16.626980] --iova-mode=pa '
                                            '[2022-10-25 23:10:16.626987] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:10:16.626995] '
                                            '--match-allocations [2022-10-25 '
                                            '23:10:16.627002] '
                                            '--file-prefix=spdk_pid83624 '
                                            '[2022-10-25 23:10:16.627009] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:10:16.765360] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:10:16.817512] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:10:16.818222] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.output'),
                                  'state': {'begin': 1666725126.2161548,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.227761268615723,
                                            'end': 1666725137.443916,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_62.state')},
                       'cmd_63': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.output'),
                                  'state': {'begin': 1666725137.446754,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7797489166259766,
                                            'end': 1666725138.226503,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_63.state')},
                       'cmd_64': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '216380.02 IOPS     845.23 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '216380.02 IOPS     845.23 MiB/s\n'
                                            '[2022-10-25 23:10:29.570713] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:10:29.570782] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:10:29.570792] bdevperf '
                                            '[2022-10-25 23:10:29.570801] '
                                            '--no-shconf [2022-10-25 '
                                            '23:10:29.570810] -l 0 [2022-10-25 '
                                            '23:10:29.570818] --huge-unlink '
                                            '[2022-10-25 23:10:29.570826] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:10:29.570834] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:10:29.570842] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:10:29.570850] --iova-mode=pa '
                                            '[2022-10-25 23:10:29.570859] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:10:29.570867] '
                                            '--match-allocations [2022-10-25 '
                                            '23:10:29.570876] '
                                            '--file-prefix=spdk_pid83649 '
                                            '[2022-10-25 23:10:29.570884] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:10:29.709427] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:10:29.753600] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:10:29.754303] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.output'),
                                  'state': {'begin': 1666725139.2636514,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.116545677185059,
                                            'end': 1666725150.380197,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.output'),
                                  'state': {'begin': 1666725150.3830373,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7887518405914307,
                                            'end': 1666725151.1717892,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_65.state')},
                       'cmd_66': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '216607.90 IOPS     846.12 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '216607.90 IOPS     846.12 MiB/s\n'
                                            '[2022-10-25 23:10:42.527926] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:10:42.527995] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:10:42.528006] bdevperf '
                                            '[2022-10-25 23:10:42.528015] '
                                            '--no-shconf [2022-10-25 '
                                            '23:10:42.528023] -l 0 [2022-10-25 '
                                            '23:10:42.528030] --huge-unlink '
                                            '[2022-10-25 23:10:42.528039] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:10:42.528047] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:10:42.528054] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:10:42.528080] --iova-mode=pa '
                                            '[2022-10-25 23:10:42.528088] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:10:42.528096] '
                                            '--match-allocations [2022-10-25 '
                                            '23:10:42.528104] '
                                            '--file-prefix=spdk_pid83674 '
                                            '[2022-10-25 23:10:42.528119] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:10:42.666440] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:10:42.709521] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:10:42.710222] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.output'),
                                  'state': {'begin': 1666725152.213005,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.126810789108276,
                                            'end': 1666725163.3398159,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_66.state')},
                       'cmd_67': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.output'),
                                  'state': {'begin': 1666725163.3429255,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.789379358291626,
                                            'end': 1666725164.132305,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_67.state')},
                       'cmd_68': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '209760.64 IOPS     819.38 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '209760.64 IOPS     819.38 MiB/s\n'
                                            '[2022-10-25 23:10:55.470262] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:10:55.470332] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:10:55.470343] bdevperf '
                                            '[2022-10-25 23:10:55.470352] '
                                            '--no-shconf [2022-10-25 '
                                            '23:10:55.470361] -l 0 [2022-10-25 '
                                            '23:10:55.470369] --huge-unlink '
                                            '[2022-10-25 23:10:55.470378] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:10:55.470386] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:10:55.470393] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:10:55.470402] --iova-mode=pa '
                                            '[2022-10-25 23:10:55.470409] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:10:55.470417] '
                                            '--match-allocations [2022-10-25 '
                                            '23:10:55.470425] '
                                            '--file-prefix=spdk_pid83699 '
                                            '[2022-10-25 23:10:55.470433] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:10:55.609173] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:10:55.669534] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:10:55.670217] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.output'),
                                  'state': {'begin': 1666725165.1595466,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.120136499404907,
                                            'end': 1666725176.279683,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_68.state')},
                       'cmd_69': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.output'),
                                  'state': {'begin': 1666725176.28263,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7875056266784668,
                                            'end': 1666725177.0701356,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_69.state')},
                       'cmd_70': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '209376.05 IOPS     817.88 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '209376.05 IOPS     817.88 MiB/s\n'
                                            '[2022-10-25 23:11:08.404264] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:11:08.404334] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:11:08.404345] bdevperf '
                                            '[2022-10-25 23:11:08.404354] '
                                            '--no-shconf [2022-10-25 '
                                            '23:11:08.404362] -l 0 [2022-10-25 '
                                            '23:11:08.404370] --huge-unlink '
                                            '[2022-10-25 23:11:08.404379] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:11:08.404386] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:11:08.404394] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:11:08.404402] --iova-mode=pa '
                                            '[2022-10-25 23:11:08.404410] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:11:08.404418] '
                                            '--match-allocations [2022-10-25 '
                                            '23:11:08.404426] '
                                            '--file-prefix=spdk_pid83724 '
                                            '[2022-10-25 23:11:08.404434] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:11:08.542984] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:11:08.589571] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:11:08.590268] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.output'),
                                  'state': {'begin': 1666725178.096931,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.102919816970825,
                                            'end': 1666725189.1998508,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.output'),
                                  'state': {'begin': 1666725189.2025259,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8047060966491699,
                                            'end': 1666725190.007232,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_71.state')},
                       'cmd_72': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '209107.46 IOPS     816.83 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '209107.46 IOPS     816.83 MiB/s\n'
                                            '[2022-10-25 23:11:21.347093] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:11:21.347171] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:11:21.347182] bdevperf '
                                            '[2022-10-25 23:11:21.347191] '
                                            '--no-shconf [2022-10-25 '
                                            '23:11:21.347200] -l 0 [2022-10-25 '
                                            '23:11:21.347208] --huge-unlink '
                                            '[2022-10-25 23:11:21.347217] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:11:21.347225] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:11:21.347234] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:11:21.347242] --iova-mode=pa '
                                            '[2022-10-25 23:11:21.347250] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:11:21.347258] '
                                            '--match-allocations [2022-10-25 '
                                            '23:11:21.347266] '
                                            '--file-prefix=spdk_pid83749 '
                                            '[2022-10-25 23:11:21.347274] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:11:21.485620] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:11:21.541528] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:11:21.542206] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.output'),
                                  'state': {'begin': 1666725191.0355625,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.117104768753052,
                                            'end': 1666725202.1526673,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_72.state')},
                       'cmd_73': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.output'),
                                  'state': {'begin': 1666725202.1556916,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7884480953216553,
                                            'end': 1666725202.9441397,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_73.state')},
                       'cmd_74': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '221094.82 IOPS     863.65 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '221094.82 IOPS     863.65 MiB/s\n'
                                            '[2022-10-25 23:11:34.290214] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:11:34.290284] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:11:34.290294] bdevperf '
                                            '[2022-10-25 23:11:34.290303] '
                                            '--no-shconf [2022-10-25 '
                                            '23:11:34.290311] -l 0 [2022-10-25 '
                                            '23:11:34.290319] --huge-unlink '
                                            '[2022-10-25 23:11:34.290326] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:11:34.290335] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:11:34.290342] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:11:34.290350] --iova-mode=pa '
                                            '[2022-10-25 23:11:34.290376] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:11:34.290385] '
                                            '--match-allocations [2022-10-25 '
                                            '23:11:34.290393] '
                                            '--file-prefix=spdk_pid83774 '
                                            '[2022-10-25 23:11:34.290409] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:11:34.428902] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:11:34.469602] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:11:34.470285] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.output'),
                                  'state': {'begin': 1666725203.9746785,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.109240770339966,
                                            'end': 1666725215.0839193,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_74.state')},
                       'cmd_75': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.output'),
                                  'state': {'begin': 1666725215.0867696,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7892894744873047,
                                            'end': 1666725215.876059,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_75.state')},
                       'cmd_76': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '221310.00 IOPS     864.49 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '221310.00 IOPS     864.49 MiB/s\n'
                                            '[2022-10-25 23:11:47.217369] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:11:47.217440] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:11:47.217451] bdevperf '
                                            '[2022-10-25 23:11:47.217461] '
                                            '--no-shconf [2022-10-25 '
                                            '23:11:47.217468] -l 0 [2022-10-25 '
                                            '23:11:47.217476] --huge-unlink '
                                            '[2022-10-25 23:11:47.217484] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:11:47.217491] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:11:47.217498] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:11:47.217505] --iova-mode=pa '
                                            '[2022-10-25 23:11:47.217513] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:11:47.217520] '
                                            '--match-allocations [2022-10-25 '
                                            '23:11:47.217527] '
                                            '--file-prefix=spdk_pid83799 '
                                            '[2022-10-25 23:11:47.217534] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:11:47.355898] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:11:47.397531] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:11:47.398212] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.output'),
                                  'state': {'begin': 1666725216.9129891,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.102845430374146,
                                            'end': 1666725228.0158346,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_76.state')},
                       'cmd_77': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.output'),
                                  'state': {'begin': 1666725228.0186582,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7831084728240967,
                                            'end': 1666725228.8017666,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_77.state')},
                       'cmd_78': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '221270.62 IOPS     864.34 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '221270.62 IOPS     864.34 MiB/s\n'
                                            '[2022-10-25 23:12:00.149272] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:12:00.149341] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:12:00.149351] bdevperf '
                                            '[2022-10-25 23:12:00.149360] '
                                            '--no-shconf [2022-10-25 '
                                            '23:12:00.149369] -l 0 [2022-10-25 '
                                            '23:12:00.149377] --huge-unlink '
                                            '[2022-10-25 23:12:00.149386] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:12:00.149393] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:12:00.149401] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:12:00.149409] --iova-mode=pa '
                                            '[2022-10-25 23:12:00.149418] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:12:00.149426] '
                                            '--match-allocations [2022-10-25 '
                                            '23:12:00.149434] '
                                            '--file-prefix=spdk_pid83825 '
                                            '[2022-10-25 23:12:00.149442] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:12:00.287764] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:12:00.329617] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:12:00.330281] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.output'),
                                  'state': {'begin': 1666725229.8440409,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.098433256149292,
                                            'end': 1666725240.9424741,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_78.state')},
                       'cmd_79': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.output'),
                                  'state': {'begin': 1666725240.9454057,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7916970252990723,
                                            'end': 1666725241.7371027,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_79.state')},
                       'cmd_80': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '219908.59 IOPS     859.02 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '219908.59 IOPS     859.02 MiB/s\n'
                                            '[2022-10-25 23:12:13.069798] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:12:13.069876] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:12:13.069887] bdevperf '
                                            '[2022-10-25 23:12:13.069896] '
                                            '--no-shconf [2022-10-25 '
                                            '23:12:13.069905] -l 0 [2022-10-25 '
                                            '23:12:13.069912] --huge-unlink '
                                            '[2022-10-25 23:12:13.069921] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:12:13.069929] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:12:13.069936] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:12:13.069945] --iova-mode=pa '
                                            '[2022-10-25 23:12:13.069953] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:12:13.069961] '
                                            '--match-allocations [2022-10-25 '
                                            '23:12:13.069969] '
                                            '--file-prefix=spdk_pid83850 '
                                            '[2022-10-25 23:12:13.069977] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:12:13.208316] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:12:13.249582] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:12:13.250261] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.output'),
                                  'state': {'begin': 1666725242.7690065,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.156771421432495,
                                            'end': 1666725253.925778,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_80.state')},
                       'cmd_81': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.output'),
                                  'state': {'begin': 1666725253.9289076,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7852492332458496,
                                            'end': 1666725254.7141569,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_81.state')},
                       'cmd_82': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '220284.56 IOPS     860.49 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '220284.56 IOPS     860.49 MiB/s\n'
                                            '[2022-10-25 23:12:26.055510] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:12:26.055579] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:12:26.055589] bdevperf '
                                            '[2022-10-25 23:12:26.055598] '
                                            '--no-shconf [2022-10-25 '
                                            '23:12:26.055606] -l 0 [2022-10-25 '
                                            '23:12:26.055614] --huge-unlink '
                                            '[2022-10-25 23:12:26.055621] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:12:26.055628] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:12:26.055636] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:12:26.055643] --iova-mode=pa '
                                            '[2022-10-25 23:12:26.055650] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:12:26.055657] '
                                            '--match-allocations [2022-10-25 '
                                            '23:12:26.055665] '
                                            '--file-prefix=spdk_pid83875 '
                                            '[2022-10-25 23:12:26.055672] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:12:26.194021] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:12:26.245587] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:12:26.246263] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.output'),
                                  'state': {'begin': 1666725255.74967,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.196541547775269,
                                            'end': 1666725266.9462116,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_82.state')},
                       'cmd_83': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.output'),
                                  'state': {'begin': 1666725266.949083,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7843360900878906,
                                            'end': 1666725267.7334192,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_83.state')},
                       'cmd_84': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '220435.91 IOPS     861.08 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '220435.91 IOPS     861.08 MiB/s\n'
                                            '[2022-10-25 23:12:39.062862] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:12:39.062933] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:12:39.062944] bdevperf '
                                            '[2022-10-25 23:12:39.062952] '
                                            '--no-shconf [2022-10-25 '
                                            '23:12:39.062961] -l 0 [2022-10-25 '
                                            '23:12:39.062969] --huge-unlink '
                                            '[2022-10-25 23:12:39.062978] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:12:39.062985] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:12:39.062992] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:12:39.063000] --iova-mode=pa '
                                            '[2022-10-25 23:12:39.063008] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:12:39.063016] '
                                            '--match-allocations [2022-10-25 '
                                            '23:12:39.063024] '
                                            '--file-prefix=spdk_pid83901 '
                                            '[2022-10-25 23:12:39.063033] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:12:39.201528] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:12:39.258472] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:12:39.259141] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.output'),
                                  'state': {'begin': 1666725268.7572954,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_aio_foo.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.184575080871582,
                                            'end': 1666725279.9418705,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_84.state')},
                       'cmd_85': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.output'),
                                  'state': {'begin': 1666725279.944596,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7948479652404785,
                                            'end': 1666725280.739444,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_85.state')},
                       'cmd_86': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '218110.54 IOPS     851.99 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '218110.54 IOPS     851.99 MiB/s\n'
                                            '[2022-10-25 23:12:52.069012] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:12:52.069081] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:12:52.069091] bdevperf '
                                            '[2022-10-25 23:12:52.069100] '
                                            '--no-shconf [2022-10-25 '
                                            '23:12:52.069108] -l 0 [2022-10-25 '
                                            '23:12:52.069116] --huge-unlink '
                                            '[2022-10-25 23:12:52.069124] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:12:52.069131] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:12:52.069139] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:12:52.069146] --iova-mode=pa '
                                            '[2022-10-25 23:12:52.069153] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:12:52.069161] '
                                            '--match-allocations [2022-10-25 '
                                            '23:12:52.069168] '
                                            '--file-prefix=spdk_pid83926 '
                                            '[2022-10-25 23:12:52.069176] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:12:52.207851] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:12:52.253574] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:12:52.254257] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.output'),
                                  'state': {'begin': 1666725281.7638643,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.18337607383728,
                                            'end': 1666725292.9472404,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_86.state')},
                       'cmd_87': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.output'),
                                  'state': {'begin': 1666725292.9499643,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7913548946380615,
                                            'end': 1666725293.7413192,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_87.state')},
                       'cmd_88': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '217774.07 IOPS     850.68 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '217774.07 IOPS     850.68 MiB/s\n'
                                            '[2022-10-25 23:13:05.092769] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:13:05.092840] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:13:05.092850] bdevperf '
                                            '[2022-10-25 23:13:05.092859] '
                                            '--no-shconf [2022-10-25 '
                                            '23:13:05.092868] -l 0 [2022-10-25 '
                                            '23:13:05.092876] --huge-unlink '
                                            '[2022-10-25 23:13:05.092885] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:13:05.092894] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:13:05.092901] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:13:05.092909] --iova-mode=pa '
                                            '[2022-10-25 23:13:05.092918] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:13:05.092926] '
                                            '--match-allocations [2022-10-25 '
                                            '23:13:05.092934] '
                                            '--file-prefix=spdk_pid83952 '
                                            '[2022-10-25 23:13:05.092943] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:13:05.231410] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:13:05.285594] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:13:05.286255] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.output'),
                                  'state': {'begin': 1666725294.7792244,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.191197872161865,
                                            'end': 1666725305.9704223,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_88.state')},
                       'cmd_89': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.output'),
                                  'state': {'begin': 1666725305.9733083,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8015813827514648,
                                            'end': 1666725306.7748897,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_89.state')},
                       'cmd_90': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '217800.45 IOPS     850.78 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '217800.45 IOPS     850.78 MiB/s\n'
                                            '[2022-10-25 23:13:18.110405] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:13:18.110473] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:13:18.110484] bdevperf '
                                            '[2022-10-25 23:13:18.110492] '
                                            '--no-shconf [2022-10-25 '
                                            '23:13:18.110500] -l 0 [2022-10-25 '
                                            '23:13:18.110508] --huge-unlink '
                                            '[2022-10-25 23:13:18.110515] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:13:18.110523] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:13:18.110530] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:13:18.110538] --iova-mode=pa '
                                            '[2022-10-25 23:13:18.110544] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:13:18.110552] '
                                            '--match-allocations [2022-10-25 '
                                            '23:13:18.110559] '
                                            '--file-prefix=spdk_pid83977 '
                                            '[2022-10-25 23:13:18.110566] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:13:18.249238] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:13:18.293492] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:13:18.294175] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.output'),
                                  'state': {'begin': 1666725307.8022592,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_libaio.conf '
                                                   '-q 4 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.190491676330566,
                                            'end': 1666725318.992751,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_90.state')},
                       'cmd_91': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.output'),
                                  'state': {'begin': 1666725319.0001276,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7905046939849854,
                                            'end': 1666725319.7906322,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_91.state')},
                       'cmd_92': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :       '
                                            '0.06 IOPS       0.00 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       '
                                            ':       0.06 IOPS       0.00 '
                                            'MiB/s\n'
                                            '[2022-10-25 23:13:31.240110] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:13:31.240182] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:13:31.240193] bdevperf '
                                            '[2022-10-25 23:13:31.240202] '
                                            '--no-shconf [2022-10-25 '
                                            '23:13:31.240210] -l 0 [2022-10-25 '
                                            '23:13:31.240219] --huge-unlink '
                                            '[2022-10-25 23:13:31.240227] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:13:31.240234] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:13:31.240241] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:13:31.240249] --iova-mode=pa '
                                            '[2022-10-25 23:13:31.240257] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:13:31.240265] '
                                            '--match-allocations [2022-10-25 '
                                            '23:13:31.240272] '
                                            '--file-prefix=spdk_pid84002 '
                                            '[2022-10-25 23:13:31.240279] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:13:31.378840] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:13:31.433486] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:13:31.434166] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.output'),
                                  'state': {'begin': 1666725320.8388972,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 398.423148393631,
                                            'end': 1666725719.2620456,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_92.state')},
                       'cmd_93': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.output'),
                                  'state': {'begin': 1666725719.2653906,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.8903448581695557,
                                            'end': 1666725720.1557355,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_93.state')},
                       'cmd_94': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '222460.54 IOPS     868.99 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '222460.54 IOPS     868.99 MiB/s\n'
                                            '[2022-10-25 23:20:11.844287] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:20:11.844351] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:20:11.844362] bdevperf '
                                            '[2022-10-25 23:20:11.844371] '
                                            '--no-shconf [2022-10-25 '
                                            '23:20:11.844380] -l 0 [2022-10-25 '
                                            '23:20:11.844388] --huge-unlink '
                                            '[2022-10-25 23:20:11.844396] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:20:11.844405] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:20:11.844412] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:20:11.844438] --iova-mode=pa '
                                            '[2022-10-25 23:20:11.844448] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:20:11.844456] '
                                            '--match-allocations [2022-10-25 '
                                            '23:20:11.844465] '
                                            '--file-prefix=spdk_pid84034 '
                                            '[2022-10-25 23:20:11.844481] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:20:11.982803] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:20:12.037539] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:20:12.038213] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.output'),
                                  'state': {'begin': 1666725721.347637,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.330201387405396,
                                            'end': 1666725732.6778383,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.output'),
                                  'state': {'begin': 1666725732.6809075,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.786815881729126,
                                            'end': 1666725733.4677234,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_95.state')},
                       'cmd_96': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '222470.91 IOPS     869.03 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '222470.91 IOPS     869.03 MiB/s\n'
                                            '[2022-10-25 23:20:24.779813] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:20:24.779881] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:20:24.779892] bdevperf '
                                            '[2022-10-25 23:20:24.779901] '
                                            '--no-shconf [2022-10-25 '
                                            '23:20:24.779910] -l 0 [2022-10-25 '
                                            '23:20:24.779918] --huge-unlink '
                                            '[2022-10-25 23:20:24.779927] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:20:24.779934] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:20:24.779942] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:20:24.779950] --iova-mode=pa '
                                            '[2022-10-25 23:20:24.779958] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:20:24.779966] '
                                            '--match-allocations [2022-10-25 '
                                            '23:20:24.779974] '
                                            '--file-prefix=spdk_pid84061 '
                                            '[2022-10-25 23:20:24.779983] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:20:24.918965] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:20:24.961539] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:20:24.962206] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.output'),
                                  'state': {'begin': 1666725734.491195,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_xnvme_io_uring_cmd.conf '
                                                   '-q 8 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.11571478843689,
                                            'end': 1666725745.6069098,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_96.state')},
                       'cmd_97': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.output'),
                                  'state': {'begin': 1666725745.6098454,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7851316928863525,
                                            'end': 1666725746.394977,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_97.state')},
                       'cmd_98': {'output': 'Running I/O for 10 seconds...\n'
                                            '\n'
                                            ' Job: Nvme0n1 (Core Mask 0x1)\n'
                                            '\t Nvme0n1             :  '
                                            '223754.18 IOPS     874.04 MiB/s\n'
                                            '\n'
                                            ' '
                                            '=============================================================\n'
                                            '\n'
                                            ' Total                       :  '
                                            '223754.18 IOPS     874.04 MiB/s\n'
                                            '[2022-10-25 23:20:37.720662] '
                                            'Starting SPDK v23.01-pre git sha1 '
                                            'fd41a2937 / DPDK 22.07.0 '
                                            'initialization...\n'
                                            '[2022-10-25 23:20:37.720732] [ '
                                            'DPDK EAL parameters: [2022-10-25 '
                                            '23:20:37.720742] bdevperf '
                                            '[2022-10-25 23:20:37.720751] '
                                            '--no-shconf [2022-10-25 '
                                            '23:20:37.720760] -l 0 [2022-10-25 '
                                            '23:20:37.720768] --huge-unlink '
                                            '[2022-10-25 23:20:37.720776] '
                                            '--log-level=lib.eal:6 [2022-10-25 '
                                            '23:20:37.720784] '
                                            '--log-level=lib.cryptodev:5 '
                                            '[2022-10-25 23:20:37.720792] '
                                            '--log-level=user1:6 [2022-10-25 '
                                            '23:20:37.720800] --iova-mode=pa '
                                            '[2022-10-25 23:20:37.720809] '
                                            '--base-virtaddr=0x200000000000 '
                                            '[2022-10-25 23:20:37.720817] '
                                            '--match-allocations [2022-10-25 '
                                            '23:20:37.720825] '
                                            '--file-prefix=spdk_pid84086 '
                                            '[2022-10-25 23:20:37.720833] ]\n'
                                            'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n'
                                            '[2022-10-25 23:20:37.859472] '
                                            'app.c: 586:spdk_app_start: '
                                            '*NOTICE*: Total cores available: '
                                            '1\n'
                                            '[2022-10-25 23:20:37.909520] '
                                            'reactor.c: 926:reactor_run: '
                                            '*NOTICE*: Reactor started on core '
                                            '0\n'
                                            '[2022-10-25 23:20:37.910195] '
                                            'accel_sw.c: '
                                            '461:sw_accel_module_init: '
                                            '*NOTICE*: Accel framework '
                                            'software module initialized.\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.output'),
                                  'state': {'begin': 1666725747.426107,
                                            'cmd': '/root/git/spdk/test/bdev/bdevperf/bdevperf '
                                                   '--json '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '-q 8 -o 4096 -w randread '
                                                   "-t 10 -m '[0]'",
                                            'cwd': 'None',
                                            'elapsed': 11.111924409866333,
                                            'end': 1666725758.5380313,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_98.state')},
                       'cmd_99': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.output'),
                                  'state': {'begin': 1666725758.540498,
                                            'cmd': 'rm '
                                                   '/tmp/spdk_bdev_uring_foo.conf '
                                                   '|| true',
                                            'cwd': 'None',
                                            'elapsed': 0.7880303859710693,
                                            'end': 1666725759.3285284,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/001_bdevperf/cmd_99.state')}}},
 'id': '001_bdevperf',
 'name': 'bdevperf',
 'nr': 1,
 'status': {'elapsed': 1166.063898563385,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'bdevperf',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['4K'], 'repetitions': 3}}
$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   96312.06 IOPS     376.22 MiB/s

 =============================================================

 Total                       :   96312.06 IOPS     376.22 MiB/s
[2022-10-25 23:03:47.429048] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:03:47.429120] [ DPDK EAL parameters: [2022-10-25 23:03:47.429132] bdevperf [2022-10-25 23:03:47.429142] --no-shconf [2022-10-25 23:03:47.429151] -l 0 [2022-10-25 23:03:47.429169] --huge-unlink [2022-10-25 23:03:47.429179] --log-level=lib.eal:6 [2022-10-25 23:03:47.429188] --log-level=lib.cryptodev:5 [2022-10-25 23:03:47.429196] --log-level=user1:6 [2022-10-25 23:03:47.429205] --iova-mode=pa [2022-10-25 23:03:47.429219] --base-virtaddr=0x200000000000 [2022-10-25 23:03:47.429227] --match-allocations [2022-10-25 23:03:47.429235] --file-prefix=spdk_pid82863 [2022-10-25 23:03:47.429243] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:03:47.567686] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:03:47.609539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:03:47.610229] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   91332.77 IOPS     356.77 MiB/s

 =============================================================

 Total                       :   91332.77 IOPS     356.77 MiB/s
[2022-10-25 23:04:00.370178] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:00.370248] [ DPDK EAL parameters: [2022-10-25 23:04:00.370259] bdevperf [2022-10-25 23:04:00.370268] --no-shconf [2022-10-25 23:04:00.370277] -l 0 [2022-10-25 23:04:00.370285] --huge-unlink [2022-10-25 23:04:00.370293] --log-level=lib.eal:6 [2022-10-25 23:04:00.370302] --log-level=lib.cryptodev:5 [2022-10-25 23:04:00.370309] --log-level=user1:6 [2022-10-25 23:04:00.370317] --iova-mode=pa [2022-10-25 23:04:00.370325] --base-virtaddr=0x200000000000 [2022-10-25 23:04:00.370333] --match-allocations [2022-10-25 23:04:00.370341] --file-prefix=spdk_pid82888 [2022-10-25 23:04:00.370350] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:00.508790] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:00.557574] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:00.558236] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   91051.09 IOPS     355.67 MiB/s

 =============================================================

 Total                       :   91051.09 IOPS     355.67 MiB/s
[2022-10-25 23:04:13.314791] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:13.314863] [ DPDK EAL parameters: [2022-10-25 23:04:13.314874] bdevperf [2022-10-25 23:04:13.314882] --no-shconf [2022-10-25 23:04:13.314890] -l 0 [2022-10-25 23:04:13.314898] --huge-unlink [2022-10-25 23:04:13.314905] --log-level=lib.eal:6 [2022-10-25 23:04:13.314913] --log-level=lib.cryptodev:5 [2022-10-25 23:04:13.314920] --log-level=user1:6 [2022-10-25 23:04:13.314927] --iova-mode=pa [2022-10-25 23:04:13.314935] --base-virtaddr=0x200000000000 [2022-10-25 23:04:13.314942] --match-allocations [2022-10-25 23:04:13.314949] --file-prefix=spdk_pid82913 [2022-10-25 23:04:13.314956] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:13.453309] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:13.493561] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:13.494240] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_uring_foo.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   87268.14 IOPS     340.89 MiB/s

 =============================================================

 Total                       :   87268.14 IOPS     340.89 MiB/s
[2022-10-25 23:04:26.232081] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:26.232151] [ DPDK EAL parameters: [2022-10-25 23:04:26.232162] bdevperf [2022-10-25 23:04:26.232171] --no-shconf [2022-10-25 23:04:26.232180] -l 0 [2022-10-25 23:04:26.232189] --huge-unlink [2022-10-25 23:04:26.232197] --log-level=lib.eal:6 [2022-10-25 23:04:26.232205] --log-level=lib.cryptodev:5 [2022-10-25 23:04:26.232212] --log-level=user1:6 [2022-10-25 23:04:26.232221] --iova-mode=pa [2022-10-25 23:04:26.232229] --base-virtaddr=0x200000000000 [2022-10-25 23:04:26.232237] --match-allocations [2022-10-25 23:04:26.232245] --file-prefix=spdk_pid82940 [2022-10-25 23:04:26.232253] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:26.370496] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:26.425602] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:26.426289] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   87338.70 IOPS     341.17 MiB/s

 =============================================================

 Total                       :   87338.70 IOPS     341.17 MiB/s
[2022-10-25 23:04:39.195818] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:39.195889] [ DPDK EAL parameters: [2022-10-25 23:04:39.195900] bdevperf [2022-10-25 23:04:39.195909] --no-shconf [2022-10-25 23:04:39.195918] -l 0 [2022-10-25 23:04:39.195926] --huge-unlink [2022-10-25 23:04:39.195934] --log-level=lib.eal:6 [2022-10-25 23:04:39.195942] --log-level=lib.cryptodev:5 [2022-10-25 23:04:39.195949] --log-level=user1:6 [2022-10-25 23:04:39.195957] --iova-mode=pa [2022-10-25 23:04:39.195966] --base-virtaddr=0x200000000000 [2022-10-25 23:04:39.195973] --match-allocations [2022-10-25 23:04:39.195981] --file-prefix=spdk_pid82965 [2022-10-25 23:04:39.195989] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:39.333951] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:39.381531] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:39.382192] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223720.83 IOPS     873.91 MiB/s

 =============================================================

 Total                       :  223720.83 IOPS     873.91 MiB/s
[2022-10-25 23:20:50.653460] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:50.653531] [ DPDK EAL parameters: [2022-10-25 23:20:50.653542] bdevperf [2022-10-25 23:20:50.653551] --no-shconf [2022-10-25 23:20:50.653560] -l 0 [2022-10-25 23:20:50.653567] --huge-unlink [2022-10-25 23:20:50.653575] --log-level=lib.eal:6 [2022-10-25 23:20:50.653583] --log-level=lib.cryptodev:5 [2022-10-25 23:20:50.653591] --log-level=user1:6 [2022-10-25 23:20:50.653599] --iova-mode=pa [2022-10-25 23:20:50.653607] --base-virtaddr=0x200000000000 [2022-10-25 23:20:50.653614] --match-allocations [2022-10-25 23:20:50.653622] --file-prefix=spdk_pid84111 [2022-10-25 23:20:50.653629] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:50.792163] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:50.833572] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:50.834238] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223708.71 IOPS     873.86 MiB/s

 =============================================================

 Total                       :  223708.71 IOPS     873.86 MiB/s
[2022-10-25 23:21:03.647189] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:03.647258] [ DPDK EAL parameters: [2022-10-25 23:21:03.647268] bdevperf [2022-10-25 23:21:03.647277] --no-shconf [2022-10-25 23:21:03.647286] -l 0 [2022-10-25 23:21:03.647295] --huge-unlink [2022-10-25 23:21:03.647303] --log-level=lib.eal:6 [2022-10-25 23:21:03.647311] --log-level=lib.cryptodev:5 [2022-10-25 23:21:03.647318] --log-level=user1:6 [2022-10-25 23:21:03.647326] --iova-mode=pa [2022-10-25 23:21:03.647334] --base-virtaddr=0x200000000000 [2022-10-25 23:21:03.647342] --match-allocations [2022-10-25 23:21:03.647350] --file-prefix=spdk_pid84137 [2022-10-25 23:21:03.647359] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:03.785565] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:03.829482] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:03.830158] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223714.32 IOPS     873.88 MiB/s

 =============================================================

 Total                       :  223714.32 IOPS     873.88 MiB/s
[2022-10-25 23:21:16.578379] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:16.578446] [ DPDK EAL parameters: [2022-10-25 23:21:16.578457] bdevperf [2022-10-25 23:21:16.578465] --no-shconf [2022-10-25 23:21:16.578474] -l 0 [2022-10-25 23:21:16.578482] --huge-unlink [2022-10-25 23:21:16.578489] --log-level=lib.eal:6 [2022-10-25 23:21:16.578497] --log-level=lib.cryptodev:5 [2022-10-25 23:21:16.578504] --log-level=user1:6 [2022-10-25 23:21:16.578512] --iova-mode=pa [2022-10-25 23:21:16.578519] --base-virtaddr=0x200000000000 [2022-10-25 23:21:16.578526] --match-allocations [2022-10-25 23:21:16.578534] --file-prefix=spdk_pid84162 [2022-10-25 23:21:16.578541] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:16.717093] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:16.765505] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:16.766176] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223656.87 IOPS     873.66 MiB/s

 =============================================================

 Total                       :  223656.87 IOPS     873.66 MiB/s
[2022-10-25 23:21:29.521276] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:29.521350] [ DPDK EAL parameters: [2022-10-25 23:21:29.521362] bdevperf [2022-10-25 23:21:29.521371] --no-shconf [2022-10-25 23:21:29.521380] -l 0 [2022-10-25 23:21:29.521388] --huge-unlink [2022-10-25 23:21:29.521396] --log-level=lib.eal:6 [2022-10-25 23:21:29.521404] --log-level=lib.cryptodev:5 [2022-10-25 23:21:29.521412] --log-level=user1:6 [2022-10-25 23:21:29.521420] --iova-mode=pa [2022-10-25 23:21:29.521428] --base-virtaddr=0x200000000000 [2022-10-25 23:21:29.521436] --match-allocations [2022-10-25 23:21:29.521443] --file-prefix=spdk_pid84187 [2022-10-25 23:21:29.521454] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:29.660283] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:29.701545] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:29.702222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223746.27 IOPS     874.01 MiB/s

 =============================================================

 Total                       :  223746.27 IOPS     874.01 MiB/s
[2022-10-25 23:21:42.426968] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:42.427037] [ DPDK EAL parameters: [2022-10-25 23:21:42.427047] bdevperf [2022-10-25 23:21:42.427056] --no-shconf [2022-10-25 23:21:42.427063] -l 0 [2022-10-25 23:21:42.427071] --huge-unlink [2022-10-25 23:21:42.427078] --log-level=lib.eal:6 [2022-10-25 23:21:42.427086] --log-level=lib.cryptodev:5 [2022-10-25 23:21:42.427093] --log-level=user1:6 [2022-10-25 23:21:42.427100] --iova-mode=pa [2022-10-25 23:21:42.427108] --base-virtaddr=0x200000000000 [2022-10-25 23:21:42.427125] --match-allocations [2022-10-25 23:21:42.427132] --file-prefix=spdk_pid84213 [2022-10-25 23:21:42.427139] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:42.565813] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:42.613477] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:42.614155] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223741.17 IOPS     873.99 MiB/s

 =============================================================

 Total                       :  223741.17 IOPS     873.99 MiB/s
[2022-10-25 23:21:55.355943] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:21:55.356022] [ DPDK EAL parameters: [2022-10-25 23:21:55.356033] bdevperf [2022-10-25 23:21:55.356042] --no-shconf [2022-10-25 23:21:55.356050] -l 0 [2022-10-25 23:21:55.356057] --huge-unlink [2022-10-25 23:21:55.356065] --log-level=lib.eal:6 [2022-10-25 23:21:55.356072] --log-level=lib.cryptodev:5 [2022-10-25 23:21:55.356079] --log-level=user1:6 [2022-10-25 23:21:55.356087] --iova-mode=pa [2022-10-25 23:21:55.356094] --base-virtaddr=0x200000000000 [2022-10-25 23:21:55.356102] --match-allocations [2022-10-25 23:21:55.356109] --file-prefix=spdk_pid84238 [2022-10-25 23:21:55.356116] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:21:55.494518] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:21:55.549627] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:21:55.550308] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223785.38 IOPS     874.16 MiB/s

 =============================================================

 Total                       :  223785.38 IOPS     874.16 MiB/s
[2022-10-25 23:22:08.359371] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:08.359442] [ DPDK EAL parameters: [2022-10-25 23:22:08.359452] bdevperf [2022-10-25 23:22:08.359461] --no-shconf [2022-10-25 23:22:08.359470] -l 0 [2022-10-25 23:22:08.359478] --huge-unlink [2022-10-25 23:22:08.359486] --log-level=lib.eal:6 [2022-10-25 23:22:08.359494] --log-level=lib.cryptodev:5 [2022-10-25 23:22:08.359501] --log-level=user1:6 [2022-10-25 23:22:08.359511] --iova-mode=pa [2022-10-25 23:22:08.359519] --base-virtaddr=0x200000000000 [2022-10-25 23:22:08.359527] --match-allocations [2022-10-25 23:22:08.359535] --file-prefix=spdk_pid84265 [2022-10-25 23:22:08.359542] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:08.497812] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:08.545510] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:08.546171] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223763.55 IOPS     874.08 MiB/s

 =============================================================

 Total                       :  223763.55 IOPS     874.08 MiB/s
[2022-10-25 23:22:21.359973] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:21.360044] [ DPDK EAL parameters: [2022-10-25 23:22:21.360054] bdevperf [2022-10-25 23:22:21.360063] --no-shconf [2022-10-25 23:22:21.360071] -l 0 [2022-10-25 23:22:21.360079] --huge-unlink [2022-10-25 23:22:21.360087] --log-level=lib.eal:6 [2022-10-25 23:22:21.360095] --log-level=lib.cryptodev:5 [2022-10-25 23:22:21.360103] --log-level=user1:6 [2022-10-25 23:22:21.360110] --iova-mode=pa [2022-10-25 23:22:21.360118] --base-virtaddr=0x200000000000 [2022-10-25 23:22:21.360126] --match-allocations [2022-10-25 23:22:21.360134] --file-prefix=spdk_pid84290 [2022-10-25 23:22:21.360141] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:21.498321] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:21.545611] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:21.546279] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223690.52 IOPS     873.79 MiB/s

 =============================================================

 Total                       :  223690.52 IOPS     873.79 MiB/s
[2022-10-25 23:22:34.349231] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:34.349305] [ DPDK EAL parameters: [2022-10-25 23:22:34.349316] bdevperf [2022-10-25 23:22:34.349325] --no-shconf [2022-10-25 23:22:34.349333] -l 0 [2022-10-25 23:22:34.349342] --huge-unlink [2022-10-25 23:22:34.349350] --log-level=lib.eal:6 [2022-10-25 23:22:34.349358] --log-level=lib.cryptodev:5 [2022-10-25 23:22:34.349365] --log-level=user1:6 [2022-10-25 23:22:34.349374] --iova-mode=pa [2022-10-25 23:22:34.349383] --base-virtaddr=0x200000000000 [2022-10-25 23:22:34.349391] --match-allocations [2022-10-25 23:22:34.349399] --file-prefix=spdk_pid84315 [2022-10-25 23:22:34.349407] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:34.487798] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:34.529488] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:34.530150] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223783.71 IOPS     874.16 MiB/s

 =============================================================

 Total                       :  223783.71 IOPS     874.16 MiB/s
[2022-10-25 23:22:47.374691] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:22:47.374761] [ DPDK EAL parameters: [2022-10-25 23:22:47.374772] bdevperf [2022-10-25 23:22:47.374781] --no-shconf [2022-10-25 23:22:47.374789] -l 0 [2022-10-25 23:22:47.374797] --huge-unlink [2022-10-25 23:22:47.374806] --log-level=lib.eal:6 [2022-10-25 23:22:47.374814] --log-level=lib.cryptodev:5 [2022-10-25 23:22:47.374821] --log-level=user1:6 [2022-10-25 23:22:47.374847] --iova-mode=pa [2022-10-25 23:22:47.374857] --base-virtaddr=0x200000000000 [2022-10-25 23:22:47.374866] --match-allocations [2022-10-25 23:22:47.374873] --file-prefix=spdk_pid84340 [2022-10-25 23:22:47.374887] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:22:47.513230] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:22:47.553617] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:22:47.554293] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   87084.15 IOPS     340.17 MiB/s

 =============================================================

 Total                       :   87084.15 IOPS     340.17 MiB/s
[2022-10-25 23:04:52.128230] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:04:52.128295] [ DPDK EAL parameters: [2022-10-25 23:04:52.128306] bdevperf [2022-10-25 23:04:52.128315] --no-shconf [2022-10-25 23:04:52.128323] -l 0 [2022-10-25 23:04:52.128331] --huge-unlink [2022-10-25 23:04:52.128339] --log-level=lib.eal:6 [2022-10-25 23:04:52.128347] --log-level=lib.cryptodev:5 [2022-10-25 23:04:52.128355] --log-level=user1:6 [2022-10-25 23:04:52.128362] --iova-mode=pa [2022-10-25 23:04:52.128371] --base-virtaddr=0x200000000000 [2022-10-25 23:04:52.128379] --match-allocations [2022-10-25 23:04:52.128386] --file-prefix=spdk_pid82990 [2022-10-25 23:04:52.128394] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:04:52.266924] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:04:52.325526] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:04:52.326204] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223745.85 IOPS     874.01 MiB/s

 =============================================================

 Total                       :  223745.85 IOPS     874.01 MiB/s
[2022-10-25 23:23:00.358689] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:23:00.358758] [ DPDK EAL parameters: [2022-10-25 23:23:00.358768] bdevperf [2022-10-25 23:23:00.358777] --no-shconf [2022-10-25 23:23:00.358786] -l 0 [2022-10-25 23:23:00.358795] --huge-unlink [2022-10-25 23:23:00.358803] --log-level=lib.eal:6 [2022-10-25 23:23:00.358811] --log-level=lib.cryptodev:5 [2022-10-25 23:23:00.358819] --log-level=user1:6 [2022-10-25 23:23:00.358826] --iova-mode=pa [2022-10-25 23:23:00.358834] --base-virtaddr=0x200000000000 [2022-10-25 23:23:00.358842] --match-allocations [2022-10-25 23:23:00.358850] --file-prefix=spdk_pid84365 [2022-10-25 23:23:00.358858] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:23:00.497334] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:23:00.545520] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:23:00.546189] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_io_uring.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   95399.03 IOPS     372.65 MiB/s

 =============================================================

 Total                       :   95399.03 IOPS     372.65 MiB/s
[2022-10-25 23:05:05.060149] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:05.060219] [ DPDK EAL parameters: [2022-10-25 23:05:05.060231] bdevperf [2022-10-25 23:05:05.060240] --no-shconf [2022-10-25 23:05:05.060249] -l 0 [2022-10-25 23:05:05.060257] --huge-unlink [2022-10-25 23:05:05.060265] --log-level=lib.eal:6 [2022-10-25 23:05:05.060273] --log-level=lib.cryptodev:5 [2022-10-25 23:05:05.060280] --log-level=user1:6 [2022-10-25 23:05:05.060288] --iova-mode=pa [2022-10-25 23:05:05.060296] --base-virtaddr=0x200000000000 [2022-10-25 23:05:05.060304] --match-allocations [2022-10-25 23:05:05.060312] --file-prefix=spdk_pid83016 [2022-10-25 23:05:05.060321] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:05.198531] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:05.245527] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:05.246196] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   95115.58 IOPS     371.55 MiB/s

 =============================================================

 Total                       :   95115.58 IOPS     371.55 MiB/s
[2022-10-25 23:05:17.970774] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:17.970842] [ DPDK EAL parameters: [2022-10-25 23:05:17.970852] bdevperf [2022-10-25 23:05:17.970861] --no-shconf [2022-10-25 23:05:17.970869] -l 0 [2022-10-25 23:05:17.970877] --huge-unlink [2022-10-25 23:05:17.970886] --log-level=lib.eal:6 [2022-10-25 23:05:17.970894] --log-level=lib.cryptodev:5 [2022-10-25 23:05:17.970902] --log-level=user1:6 [2022-10-25 23:05:17.970910] --iova-mode=pa [2022-10-25 23:05:17.970918] --base-virtaddr=0x200000000000 [2022-10-25 23:05:17.970926] --match-allocations [2022-10-25 23:05:17.970934] --file-prefix=spdk_pid83041 [2022-10-25 23:05:17.970943] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:18.109398] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:18.153610] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:18.154319] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   95206.38 IOPS     371.90 MiB/s

 =============================================================

 Total                       :   95206.38 IOPS     371.90 MiB/s
[2022-10-25 23:05:30.898804] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:30.898875] [ DPDK EAL parameters: [2022-10-25 23:05:30.898887] bdevperf [2022-10-25 23:05:30.898896] --no-shconf [2022-10-25 23:05:30.898905] -l 0 [2022-10-25 23:05:30.898912] --huge-unlink [2022-10-25 23:05:30.898921] --log-level=lib.eal:6 [2022-10-25 23:05:30.898929] --log-level=lib.cryptodev:5 [2022-10-25 23:05:30.898937] --log-level=user1:6 [2022-10-25 23:05:30.898945] --iova-mode=pa [2022-10-25 23:05:30.898953] --base-virtaddr=0x200000000000 [2022-10-25 23:05:30.898960] --match-allocations [2022-10-25 23:05:30.898968] --file-prefix=spdk_pid83066 [2022-10-25 23:05:30.898976] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:31.037583] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:31.081613] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:31.082310] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
rm: cannot remove '/tmp/spdk_bdev_aio_foo.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   94374.27 IOPS     368.65 MiB/s

 =============================================================

 Total                       :   94374.27 IOPS     368.65 MiB/s
[2022-10-25 23:05:43.831772] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:43.831843] [ DPDK EAL parameters: [2022-10-25 23:05:43.831854] bdevperf [2022-10-25 23:05:43.831863] --no-shconf [2022-10-25 23:05:43.831872] -l 0 [2022-10-25 23:05:43.831880] --huge-unlink [2022-10-25 23:05:43.831888] --log-level=lib.eal:6 [2022-10-25 23:05:43.831897] --log-level=lib.cryptodev:5 [2022-10-25 23:05:43.831905] --log-level=user1:6 [2022-10-25 23:05:43.831913] --iova-mode=pa [2022-10-25 23:05:43.831921] --base-virtaddr=0x200000000000 [2022-10-25 23:05:43.831929] --match-allocations [2022-10-25 23:05:43.831937] --file-prefix=spdk_pid83091 [2022-10-25 23:05:43.831946] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:43.970469] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:44.009497] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:44.010185] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   94300.13 IOPS     368.36 MiB/s

 =============================================================

 Total                       :   94300.13 IOPS     368.36 MiB/s
[2022-10-25 23:05:56.804496] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:05:56.804569] [ DPDK EAL parameters: [2022-10-25 23:05:56.804581] bdevperf [2022-10-25 23:05:56.804590] --no-shconf [2022-10-25 23:05:56.804599] -l 0 [2022-10-25 23:05:56.804608] --huge-unlink [2022-10-25 23:05:56.804617] --log-level=lib.eal:6 [2022-10-25 23:05:56.804625] --log-level=lib.cryptodev:5 [2022-10-25 23:05:56.804634] --log-level=user1:6 [2022-10-25 23:05:56.804643] --iova-mode=pa [2022-10-25 23:05:56.804652] --base-virtaddr=0x200000000000 [2022-10-25 23:05:56.804660] --match-allocations [2022-10-25 23:05:56.804670] --file-prefix=spdk_pid83116 [2022-10-25 23:05:56.804679] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:05:56.943818] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:05:56.989578] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:05:56.990266] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   94145.55 IOPS     367.76 MiB/s

 =============================================================

 Total                       :   94145.55 IOPS     367.76 MiB/s
[2022-10-25 23:06:09.798113] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:09.798184] [ DPDK EAL parameters: [2022-10-25 23:06:09.798194] bdevperf [2022-10-25 23:06:09.798203] --no-shconf [2022-10-25 23:06:09.798211] -l 0 [2022-10-25 23:06:09.798219] --huge-unlink [2022-10-25 23:06:09.798228] --log-level=lib.eal:6 [2022-10-25 23:06:09.798235] --log-level=lib.cryptodev:5 [2022-10-25 23:06:09.798243] --log-level=user1:6 [2022-10-25 23:06:09.798251] --iova-mode=pa [2022-10-25 23:06:09.798259] --base-virtaddr=0x200000000000 [2022-10-25 23:06:09.798267] --match-allocations [2022-10-25 23:06:09.798274] --file-prefix=spdk_pid83141 [2022-10-25 23:06:09.798282] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:09.937122] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:09.989458] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:09.990130] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
rm: cannot remove '/tmp/spdk_bdev_xnvme_libaio.conf': No such file or directory

$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   92010.24 IOPS     359.42 MiB/s

 =============================================================

 Total                       :   92010.24 IOPS     359.42 MiB/s
[2022-10-25 23:06:22.801816] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:22.801884] [ DPDK EAL parameters: [2022-10-25 23:06:22.801895] bdevperf [2022-10-25 23:06:22.801904] --no-shconf [2022-10-25 23:06:22.801913] -l 0 [2022-10-25 23:06:22.801920] --huge-unlink [2022-10-25 23:06:22.801928] --log-level=lib.eal:6 [2022-10-25 23:06:22.801936] --log-level=lib.cryptodev:5 [2022-10-25 23:06:22.801943] --log-level=user1:6 [2022-10-25 23:06:22.801968] --iova-mode=pa [2022-10-25 23:06:22.801976] --base-virtaddr=0x200000000000 [2022-10-25 23:06:22.801983] --match-allocations [2022-10-25 23:06:22.801990] --file-prefix=spdk_pid83166 [2022-10-25 23:06:22.802006] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:22.940465] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:22.981534] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:22.982224] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   92095.63 IOPS     359.75 MiB/s

 =============================================================

 Total                       :   92095.63 IOPS     359.75 MiB/s
[2022-10-25 23:06:35.798932] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:35.799005] [ DPDK EAL parameters: [2022-10-25 23:06:35.799016] bdevperf [2022-10-25 23:06:35.799025] --no-shconf [2022-10-25 23:06:35.799033] -l 0 [2022-10-25 23:06:35.799041] --huge-unlink [2022-10-25 23:06:35.799050] --log-level=lib.eal:6 [2022-10-25 23:06:35.799058] --log-level=lib.cryptodev:5 [2022-10-25 23:06:35.799065] --log-level=user1:6 [2022-10-25 23:06:35.799075] --iova-mode=pa [2022-10-25 23:06:35.799082] --base-virtaddr=0x200000000000 [2022-10-25 23:06:35.799091] --match-allocations [2022-10-25 23:06:35.799099] --file-prefix=spdk_pid83193 [2022-10-25 23:06:35.799107] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:35.937685] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:35.993482] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:35.994154] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 1 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :   91701.04 IOPS     358.21 MiB/s

 =============================================================

 Total                       :   91701.04 IOPS     358.21 MiB/s
[2022-10-25 23:06:48.806875] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:06:48.806943] [ DPDK EAL parameters: [2022-10-25 23:06:48.806953] bdevperf [2022-10-25 23:06:48.806962] --no-shconf [2022-10-25 23:06:48.806970] -l 0 [2022-10-25 23:06:48.806977] --huge-unlink [2022-10-25 23:06:48.806985] --log-level=lib.eal:6 [2022-10-25 23:06:48.806993] --log-level=lib.cryptodev:5 [2022-10-25 23:06:48.807000] --log-level=user1:6 [2022-10-25 23:06:48.807025] --iova-mode=pa [2022-10-25 23:06:48.807033] --base-virtaddr=0x200000000000 [2022-10-25 23:06:48.807041] --match-allocations [2022-10-25 23:06:48.807049] --file-prefix=spdk_pid83218 [2022-10-25 23:06:48.807064] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:06:48.945250] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:06:48.985598] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:06:48.986291] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  137677.93 IOPS     537.80 MiB/s

 =============================================================

 Total                       :  137677.93 IOPS     537.80 MiB/s
[2022-10-25 23:07:01.817511] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:01.817582] [ DPDK EAL parameters: [2022-10-25 23:07:01.817593] bdevperf [2022-10-25 23:07:01.817602] --no-shconf [2022-10-25 23:07:01.817611] -l 0 [2022-10-25 23:07:01.817618] --huge-unlink [2022-10-25 23:07:01.817628] --log-level=lib.eal:6 [2022-10-25 23:07:01.817635] --log-level=lib.cryptodev:5 [2022-10-25 23:07:01.817643] --log-level=user1:6 [2022-10-25 23:07:01.817651] --iova-mode=pa [2022-10-25 23:07:01.817659] --base-virtaddr=0x200000000000 [2022-10-25 23:07:01.817667] --match-allocations [2022-10-25 23:07:01.817675] --file-prefix=spdk_pid83243 [2022-10-25 23:07:01.817684] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:01.956216] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:02.001505] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:02.002177] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  137638.53 IOPS     537.65 MiB/s

 =============================================================

 Total                       :  137638.53 IOPS     537.65 MiB/s
[2022-10-25 23:07:14.742705] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:14.742773] [ DPDK EAL parameters: [2022-10-25 23:07:14.742783] bdevperf [2022-10-25 23:07:14.742792] --no-shconf [2022-10-25 23:07:14.742800] -l 0 [2022-10-25 23:07:14.742808] --huge-unlink [2022-10-25 23:07:14.742815] --log-level=lib.eal:6 [2022-10-25 23:07:14.742822] --log-level=lib.cryptodev:5 [2022-10-25 23:07:14.742829] --log-level=user1:6 [2022-10-25 23:07:14.742837] --iova-mode=pa [2022-10-25 23:07:14.742844] --base-virtaddr=0x200000000000 [2022-10-25 23:07:14.742851] --match-allocations [2022-10-25 23:07:14.742858] --file-prefix=spdk_pid83268 [2022-10-25 23:07:14.742866] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:14.881095] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:14.929550] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:14.930229] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  122234.74 IOPS     477.48 MiB/s

 =============================================================

 Total                       :  122234.74 IOPS     477.48 MiB/s
[2022-10-25 23:07:27.731543] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:27.731614] [ DPDK EAL parameters: [2022-10-25 23:07:27.731625] bdevperf [2022-10-25 23:07:27.731633] --no-shconf [2022-10-25 23:07:27.731642] -l 0 [2022-10-25 23:07:27.731650] --huge-unlink [2022-10-25 23:07:27.731659] --log-level=lib.eal:6 [2022-10-25 23:07:27.731666] --log-level=lib.cryptodev:5 [2022-10-25 23:07:27.731673] --log-level=user1:6 [2022-10-25 23:07:27.731681] --iova-mode=pa [2022-10-25 23:07:27.731689] --base-virtaddr=0x200000000000 [2022-10-25 23:07:27.731697] --match-allocations [2022-10-25 23:07:27.731705] --file-prefix=spdk_pid83293 [2022-10-25 23:07:27.731713] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:27.870333] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:27.917497] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:27.918172] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  123370.19 IOPS     481.91 MiB/s

 =============================================================

 Total                       :  123370.19 IOPS     481.91 MiB/s
[2022-10-25 23:07:40.684960] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:40.685028] [ DPDK EAL parameters: [2022-10-25 23:07:40.685039] bdevperf [2022-10-25 23:07:40.685048] --no-shconf [2022-10-25 23:07:40.685057] -l 0 [2022-10-25 23:07:40.685065] --huge-unlink [2022-10-25 23:07:40.685072] --log-level=lib.eal:6 [2022-10-25 23:07:40.685081] --log-level=lib.cryptodev:5 [2022-10-25 23:07:40.685088] --log-level=user1:6 [2022-10-25 23:07:40.685096] --iova-mode=pa [2022-10-25 23:07:40.685103] --base-virtaddr=0x200000000000 [2022-10-25 23:07:40.685112] --match-allocations [2022-10-25 23:07:40.685120] --file-prefix=spdk_pid83318 [2022-10-25 23:07:40.685128] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:40.823519] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:40.865686] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:40.866354] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  123229.34 IOPS     481.36 MiB/s

 =============================================================

 Total                       :  123229.34 IOPS     481.36 MiB/s
[2022-10-25 23:07:53.604306] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:07:53.604381] [ DPDK EAL parameters: [2022-10-25 23:07:53.604392] bdevperf [2022-10-25 23:07:53.604401] --no-shconf [2022-10-25 23:07:53.604410] -l 0 [2022-10-25 23:07:53.604418] --huge-unlink [2022-10-25 23:07:53.604427] --log-level=lib.eal:6 [2022-10-25 23:07:53.604434] --log-level=lib.cryptodev:5 [2022-10-25 23:07:53.604442] --log-level=user1:6 [2022-10-25 23:07:53.604450] --iova-mode=pa [2022-10-25 23:07:53.604458] --base-virtaddr=0x200000000000 [2022-10-25 23:07:53.604466] --match-allocations [2022-10-25 23:07:53.604474] --file-prefix=spdk_pid83343 [2022-10-25 23:07:53.604482] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:07:53.742580] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:07:53.813506] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:07:53.814176] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  123447.21 IOPS     482.22 MiB/s

 =============================================================

 Total                       :  123447.21 IOPS     482.22 MiB/s
[2022-10-25 23:08:06.563163] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:06.563231] [ DPDK EAL parameters: [2022-10-25 23:08:06.563241] bdevperf [2022-10-25 23:08:06.563250] --no-shconf [2022-10-25 23:08:06.563258] -l 0 [2022-10-25 23:08:06.563267] --huge-unlink [2022-10-25 23:08:06.563276] --log-level=lib.eal:6 [2022-10-25 23:08:06.563283] --log-level=lib.cryptodev:5 [2022-10-25 23:08:06.563291] --log-level=user1:6 [2022-10-25 23:08:06.563298] --iova-mode=pa [2022-10-25 23:08:06.563307] --base-virtaddr=0x200000000000 [2022-10-25 23:08:06.563315] --match-allocations [2022-10-25 23:08:06.563323] --file-prefix=spdk_pid83368 [2022-10-25 23:08:06.563331] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:06.701480] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:06.761538] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:06.762219] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  132739.00 IOPS     518.51 MiB/s

 =============================================================

 Total                       :  132739.00 IOPS     518.51 MiB/s
[2022-10-25 23:08:19.508671] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:19.508741] [ DPDK EAL parameters: [2022-10-25 23:08:19.508752] bdevperf [2022-10-25 23:08:19.508761] --no-shconf [2022-10-25 23:08:19.508770] -l 0 [2022-10-25 23:08:19.508778] --huge-unlink [2022-10-25 23:08:19.508786] --log-level=lib.eal:6 [2022-10-25 23:08:19.508793] --log-level=lib.cryptodev:5 [2022-10-25 23:08:19.508801] --log-level=user1:6 [2022-10-25 23:08:19.508808] --iova-mode=pa [2022-10-25 23:08:19.508816] --base-virtaddr=0x200000000000 [2022-10-25 23:08:19.508824] --match-allocations [2022-10-25 23:08:19.508832] --file-prefix=spdk_pid83393 [2022-10-25 23:08:19.508840] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:19.647435] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:19.689565] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:19.690233] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  132646.77 IOPS     518.15 MiB/s

 =============================================================

 Total                       :  132646.77 IOPS     518.15 MiB/s
[2022-10-25 23:08:32.474289] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:32.474360] [ DPDK EAL parameters: [2022-10-25 23:08:32.474370] bdevperf [2022-10-25 23:08:32.474379] --no-shconf [2022-10-25 23:08:32.474389] -l 0 [2022-10-25 23:08:32.474397] --huge-unlink [2022-10-25 23:08:32.474404] --log-level=lib.eal:6 [2022-10-25 23:08:32.474412] --log-level=lib.cryptodev:5 [2022-10-25 23:08:32.474419] --log-level=user1:6 [2022-10-25 23:08:32.474426] --iova-mode=pa [2022-10-25 23:08:32.474434] --base-virtaddr=0x200000000000 [2022-10-25 23:08:32.474442] --match-allocations [2022-10-25 23:08:32.474451] --file-prefix=spdk_pid83419 [2022-10-25 23:08:32.474458] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:32.612935] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:32.661606] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:32.662286] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  132765.93 IOPS     518.62 MiB/s

 =============================================================

 Total                       :  132765.93 IOPS     518.62 MiB/s
[2022-10-25 23:08:45.526319] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:45.526390] [ DPDK EAL parameters: [2022-10-25 23:08:45.526400] bdevperf [2022-10-25 23:08:45.526409] --no-shconf [2022-10-25 23:08:45.526418] -l 0 [2022-10-25 23:08:45.526426] --huge-unlink [2022-10-25 23:08:45.526433] --log-level=lib.eal:6 [2022-10-25 23:08:45.526441] --log-level=lib.cryptodev:5 [2022-10-25 23:08:45.526448] --log-level=user1:6 [2022-10-25 23:08:45.526456] --iova-mode=pa [2022-10-25 23:08:45.526464] --base-virtaddr=0x200000000000 [2022-10-25 23:08:45.526472] --match-allocations [2022-10-25 23:08:45.526479] --file-prefix=spdk_pid83445 [2022-10-25 23:08:45.526487] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:45.665426] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:45.713532] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:45.714200] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  131267.29 IOPS     512.76 MiB/s

 =============================================================

 Total                       :  131267.29 IOPS     512.76 MiB/s
[2022-10-25 23:08:58.466029] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:08:58.466097] [ DPDK EAL parameters: [2022-10-25 23:08:58.466108] bdevperf [2022-10-25 23:08:58.466116] --no-shconf [2022-10-25 23:08:58.466125] -l 0 [2022-10-25 23:08:58.466133] --huge-unlink [2022-10-25 23:08:58.466142] --log-level=lib.eal:6 [2022-10-25 23:08:58.466149] --log-level=lib.cryptodev:5 [2022-10-25 23:08:58.466157] --log-level=user1:6 [2022-10-25 23:08:58.466172] --iova-mode=pa [2022-10-25 23:08:58.466180] --base-virtaddr=0x200000000000 [2022-10-25 23:08:58.466189] --match-allocations [2022-10-25 23:08:58.466197] --file-prefix=spdk_pid83471 [2022-10-25 23:08:58.466205] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:08:58.604528] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:08:58.657517] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:08:58.658187] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  131511.69 IOPS     513.72 MiB/s

 =============================================================

 Total                       :  131511.69 IOPS     513.72 MiB/s
[2022-10-25 23:09:11.464489] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:11.464557] [ DPDK EAL parameters: [2022-10-25 23:09:11.464568] bdevperf [2022-10-25 23:09:11.464577] --no-shconf [2022-10-25 23:09:11.464586] -l 0 [2022-10-25 23:09:11.464594] --huge-unlink [2022-10-25 23:09:11.464602] --log-level=lib.eal:6 [2022-10-25 23:09:11.464611] --log-level=lib.cryptodev:5 [2022-10-25 23:09:11.464618] --log-level=user1:6 [2022-10-25 23:09:11.464626] --iova-mode=pa [2022-10-25 23:09:11.464634] --base-virtaddr=0x200000000000 [2022-10-25 23:09:11.464642] --match-allocations [2022-10-25 23:09:11.464650] --file-prefix=spdk_pid83497 [2022-10-25 23:09:11.464658] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:11.603992] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:11.645517] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:11.646198] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  131606.51 IOPS     514.09 MiB/s

 =============================================================

 Total                       :  131606.51 IOPS     514.09 MiB/s
[2022-10-25 23:09:24.448368] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:24.448436] [ DPDK EAL parameters: [2022-10-25 23:09:24.448447] bdevperf [2022-10-25 23:09:24.448456] --no-shconf [2022-10-25 23:09:24.448465] -l 0 [2022-10-25 23:09:24.448473] --huge-unlink [2022-10-25 23:09:24.448481] --log-level=lib.eal:6 [2022-10-25 23:09:24.448489] --log-level=lib.cryptodev:5 [2022-10-25 23:09:24.448496] --log-level=user1:6 [2022-10-25 23:09:24.448503] --iova-mode=pa [2022-10-25 23:09:24.448511] --base-virtaddr=0x200000000000 [2022-10-25 23:09:24.448520] --match-allocations [2022-10-25 23:09:24.448527] --file-prefix=spdk_pid83522 [2022-10-25 23:09:24.448535] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:24.587595] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:24.633513] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:24.634174] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  128907.87 IOPS     503.55 MiB/s

 =============================================================

 Total                       :  128907.87 IOPS     503.55 MiB/s
[2022-10-25 23:09:37.447829] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:37.447899] [ DPDK EAL parameters: [2022-10-25 23:09:37.447909] bdevperf [2022-10-25 23:09:37.447918] --no-shconf [2022-10-25 23:09:37.447926] -l 0 [2022-10-25 23:09:37.447934] --huge-unlink [2022-10-25 23:09:37.447943] --log-level=lib.eal:6 [2022-10-25 23:09:37.447951] --log-level=lib.cryptodev:5 [2022-10-25 23:09:37.447958] --log-level=user1:6 [2022-10-25 23:09:37.447965] --iova-mode=pa [2022-10-25 23:09:37.447973] --base-virtaddr=0x200000000000 [2022-10-25 23:09:37.447981] --match-allocations [2022-10-25 23:09:37.447988] --file-prefix=spdk_pid83547 [2022-10-25 23:09:37.447996] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:37.586338] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:37.633485] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:37.634187] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  129471.51 IOPS     505.75 MiB/s

 =============================================================

 Total                       :  129471.51 IOPS     505.75 MiB/s
[2022-10-25 23:09:50.417714] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:09:50.417786] [ DPDK EAL parameters: [2022-10-25 23:09:50.417797] bdevperf [2022-10-25 23:09:50.417806] --no-shconf [2022-10-25 23:09:50.417815] -l 0 [2022-10-25 23:09:50.417823] --huge-unlink [2022-10-25 23:09:50.417830] --log-level=lib.eal:6 [2022-10-25 23:09:50.417838] --log-level=lib.cryptodev:5 [2022-10-25 23:09:50.417845] --log-level=user1:6 [2022-10-25 23:09:50.417855] --iova-mode=pa [2022-10-25 23:09:50.417864] --base-virtaddr=0x200000000000 [2022-10-25 23:09:50.417872] --match-allocations [2022-10-25 23:09:50.417880] --file-prefix=spdk_pid83574 [2022-10-25 23:09:50.417889] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:09:50.556324] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:09:50.597526] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:09:50.598215] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 2 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  134840.52 IOPS     526.72 MiB/s

 =============================================================

 Total                       :  134840.52 IOPS     526.72 MiB/s
[2022-10-25 23:10:03.412061] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:03.412135] [ DPDK EAL parameters: [2022-10-25 23:10:03.412146] bdevperf [2022-10-25 23:10:03.412155] --no-shconf [2022-10-25 23:10:03.412164] -l 0 [2022-10-25 23:10:03.412171] --huge-unlink [2022-10-25 23:10:03.412179] --log-level=lib.eal:6 [2022-10-25 23:10:03.412187] --log-level=lib.cryptodev:5 [2022-10-25 23:10:03.412195] --log-level=user1:6 [2022-10-25 23:10:03.412204] --iova-mode=pa [2022-10-25 23:10:03.412212] --base-virtaddr=0x200000000000 [2022-10-25 23:10:03.412220] --match-allocations [2022-10-25 23:10:03.412227] --file-prefix=spdk_pid83599 [2022-10-25 23:10:03.412236] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:03.550916] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:03.597661] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:03.598365] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  220666.80 IOPS     861.98 MiB/s

 =============================================================

 Total                       :  220666.80 IOPS     861.98 MiB/s
[2022-10-25 23:10:16.626846] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:16.626916] [ DPDK EAL parameters: [2022-10-25 23:10:16.626927] bdevperf [2022-10-25 23:10:16.626935] --no-shconf [2022-10-25 23:10:16.626943] -l 0 [2022-10-25 23:10:16.626951] --huge-unlink [2022-10-25 23:10:16.626958] --log-level=lib.eal:6 [2022-10-25 23:10:16.626966] --log-level=lib.cryptodev:5 [2022-10-25 23:10:16.626973] --log-level=user1:6 [2022-10-25 23:10:16.626980] --iova-mode=pa [2022-10-25 23:10:16.626987] --base-virtaddr=0x200000000000 [2022-10-25 23:10:16.626995] --match-allocations [2022-10-25 23:10:16.627002] --file-prefix=spdk_pid83624 [2022-10-25 23:10:16.627009] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:16.765360] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:16.817512] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:16.818222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  216380.02 IOPS     845.23 MiB/s

 =============================================================

 Total                       :  216380.02 IOPS     845.23 MiB/s
[2022-10-25 23:10:29.570713] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:29.570782] [ DPDK EAL parameters: [2022-10-25 23:10:29.570792] bdevperf [2022-10-25 23:10:29.570801] --no-shconf [2022-10-25 23:10:29.570810] -l 0 [2022-10-25 23:10:29.570818] --huge-unlink [2022-10-25 23:10:29.570826] --log-level=lib.eal:6 [2022-10-25 23:10:29.570834] --log-level=lib.cryptodev:5 [2022-10-25 23:10:29.570842] --log-level=user1:6 [2022-10-25 23:10:29.570850] --iova-mode=pa [2022-10-25 23:10:29.570859] --base-virtaddr=0x200000000000 [2022-10-25 23:10:29.570867] --match-allocations [2022-10-25 23:10:29.570876] --file-prefix=spdk_pid83649 [2022-10-25 23:10:29.570884] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:29.709427] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:29.753600] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:29.754303] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  216607.90 IOPS     846.12 MiB/s

 =============================================================

 Total                       :  216607.90 IOPS     846.12 MiB/s
[2022-10-25 23:10:42.527926] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:42.527995] [ DPDK EAL parameters: [2022-10-25 23:10:42.528006] bdevperf [2022-10-25 23:10:42.528015] --no-shconf [2022-10-25 23:10:42.528023] -l 0 [2022-10-25 23:10:42.528030] --huge-unlink [2022-10-25 23:10:42.528039] --log-level=lib.eal:6 [2022-10-25 23:10:42.528047] --log-level=lib.cryptodev:5 [2022-10-25 23:10:42.528054] --log-level=user1:6 [2022-10-25 23:10:42.528080] --iova-mode=pa [2022-10-25 23:10:42.528088] --base-virtaddr=0x200000000000 [2022-10-25 23:10:42.528096] --match-allocations [2022-10-25 23:10:42.528104] --file-prefix=spdk_pid83674 [2022-10-25 23:10:42.528119] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:42.666440] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:42.709521] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:42.710222] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  209760.64 IOPS     819.38 MiB/s

 =============================================================

 Total                       :  209760.64 IOPS     819.38 MiB/s
[2022-10-25 23:10:55.470262] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:10:55.470332] [ DPDK EAL parameters: [2022-10-25 23:10:55.470343] bdevperf [2022-10-25 23:10:55.470352] --no-shconf [2022-10-25 23:10:55.470361] -l 0 [2022-10-25 23:10:55.470369] --huge-unlink [2022-10-25 23:10:55.470378] --log-level=lib.eal:6 [2022-10-25 23:10:55.470386] --log-level=lib.cryptodev:5 [2022-10-25 23:10:55.470393] --log-level=user1:6 [2022-10-25 23:10:55.470402] --iova-mode=pa [2022-10-25 23:10:55.470409] --base-virtaddr=0x200000000000 [2022-10-25 23:10:55.470417] --match-allocations [2022-10-25 23:10:55.470425] --file-prefix=spdk_pid83699 [2022-10-25 23:10:55.470433] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:10:55.609173] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:10:55.669534] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:10:55.670217] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  209376.05 IOPS     817.88 MiB/s

 =============================================================

 Total                       :  209376.05 IOPS     817.88 MiB/s
[2022-10-25 23:11:08.404264] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:08.404334] [ DPDK EAL parameters: [2022-10-25 23:11:08.404345] bdevperf [2022-10-25 23:11:08.404354] --no-shconf [2022-10-25 23:11:08.404362] -l 0 [2022-10-25 23:11:08.404370] --huge-unlink [2022-10-25 23:11:08.404379] --log-level=lib.eal:6 [2022-10-25 23:11:08.404386] --log-level=lib.cryptodev:5 [2022-10-25 23:11:08.404394] --log-level=user1:6 [2022-10-25 23:11:08.404402] --iova-mode=pa [2022-10-25 23:11:08.404410] --base-virtaddr=0x200000000000 [2022-10-25 23:11:08.404418] --match-allocations [2022-10-25 23:11:08.404426] --file-prefix=spdk_pid83724 [2022-10-25 23:11:08.404434] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:08.542984] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:08.589571] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:08.590268] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  209107.46 IOPS     816.83 MiB/s

 =============================================================

 Total                       :  209107.46 IOPS     816.83 MiB/s
[2022-10-25 23:11:21.347093] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:21.347171] [ DPDK EAL parameters: [2022-10-25 23:11:21.347182] bdevperf [2022-10-25 23:11:21.347191] --no-shconf [2022-10-25 23:11:21.347200] -l 0 [2022-10-25 23:11:21.347208] --huge-unlink [2022-10-25 23:11:21.347217] --log-level=lib.eal:6 [2022-10-25 23:11:21.347225] --log-level=lib.cryptodev:5 [2022-10-25 23:11:21.347234] --log-level=user1:6 [2022-10-25 23:11:21.347242] --iova-mode=pa [2022-10-25 23:11:21.347250] --base-virtaddr=0x200000000000 [2022-10-25 23:11:21.347258] --match-allocations [2022-10-25 23:11:21.347266] --file-prefix=spdk_pid83749 [2022-10-25 23:11:21.347274] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:21.485620] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:21.541528] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:21.542206] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  221094.82 IOPS     863.65 MiB/s

 =============================================================

 Total                       :  221094.82 IOPS     863.65 MiB/s
[2022-10-25 23:11:34.290214] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:34.290284] [ DPDK EAL parameters: [2022-10-25 23:11:34.290294] bdevperf [2022-10-25 23:11:34.290303] --no-shconf [2022-10-25 23:11:34.290311] -l 0 [2022-10-25 23:11:34.290319] --huge-unlink [2022-10-25 23:11:34.290326] --log-level=lib.eal:6 [2022-10-25 23:11:34.290335] --log-level=lib.cryptodev:5 [2022-10-25 23:11:34.290342] --log-level=user1:6 [2022-10-25 23:11:34.290350] --iova-mode=pa [2022-10-25 23:11:34.290376] --base-virtaddr=0x200000000000 [2022-10-25 23:11:34.290385] --match-allocations [2022-10-25 23:11:34.290393] --file-prefix=spdk_pid83774 [2022-10-25 23:11:34.290409] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:34.428902] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:34.469602] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:34.470285] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  221310.00 IOPS     864.49 MiB/s

 =============================================================

 Total                       :  221310.00 IOPS     864.49 MiB/s
[2022-10-25 23:11:47.217369] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:11:47.217440] [ DPDK EAL parameters: [2022-10-25 23:11:47.217451] bdevperf [2022-10-25 23:11:47.217461] --no-shconf [2022-10-25 23:11:47.217468] -l 0 [2022-10-25 23:11:47.217476] --huge-unlink [2022-10-25 23:11:47.217484] --log-level=lib.eal:6 [2022-10-25 23:11:47.217491] --log-level=lib.cryptodev:5 [2022-10-25 23:11:47.217498] --log-level=user1:6 [2022-10-25 23:11:47.217505] --iova-mode=pa [2022-10-25 23:11:47.217513] --base-virtaddr=0x200000000000 [2022-10-25 23:11:47.217520] --match-allocations [2022-10-25 23:11:47.217527] --file-prefix=spdk_pid83799 [2022-10-25 23:11:47.217534] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:11:47.355898] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:11:47.397531] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:11:47.398212] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  221270.62 IOPS     864.34 MiB/s

 =============================================================

 Total                       :  221270.62 IOPS     864.34 MiB/s
[2022-10-25 23:12:00.149272] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:00.149341] [ DPDK EAL parameters: [2022-10-25 23:12:00.149351] bdevperf [2022-10-25 23:12:00.149360] --no-shconf [2022-10-25 23:12:00.149369] -l 0 [2022-10-25 23:12:00.149377] --huge-unlink [2022-10-25 23:12:00.149386] --log-level=lib.eal:6 [2022-10-25 23:12:00.149393] --log-level=lib.cryptodev:5 [2022-10-25 23:12:00.149401] --log-level=user1:6 [2022-10-25 23:12:00.149409] --iova-mode=pa [2022-10-25 23:12:00.149418] --base-virtaddr=0x200000000000 [2022-10-25 23:12:00.149426] --match-allocations [2022-10-25 23:12:00.149434] --file-prefix=spdk_pid83825 [2022-10-25 23:12:00.149442] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:00.287764] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:00.329617] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:00.330281] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  219908.59 IOPS     859.02 MiB/s

 =============================================================

 Total                       :  219908.59 IOPS     859.02 MiB/s
[2022-10-25 23:12:13.069798] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:13.069876] [ DPDK EAL parameters: [2022-10-25 23:12:13.069887] bdevperf [2022-10-25 23:12:13.069896] --no-shconf [2022-10-25 23:12:13.069905] -l 0 [2022-10-25 23:12:13.069912] --huge-unlink [2022-10-25 23:12:13.069921] --log-level=lib.eal:6 [2022-10-25 23:12:13.069929] --log-level=lib.cryptodev:5 [2022-10-25 23:12:13.069936] --log-level=user1:6 [2022-10-25 23:12:13.069945] --iova-mode=pa [2022-10-25 23:12:13.069953] --base-virtaddr=0x200000000000 [2022-10-25 23:12:13.069961] --match-allocations [2022-10-25 23:12:13.069969] --file-prefix=spdk_pid83850 [2022-10-25 23:12:13.069977] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:13.208316] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:13.249582] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:13.250261] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  220284.56 IOPS     860.49 MiB/s

 =============================================================

 Total                       :  220284.56 IOPS     860.49 MiB/s
[2022-10-25 23:12:26.055510] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:26.055579] [ DPDK EAL parameters: [2022-10-25 23:12:26.055589] bdevperf [2022-10-25 23:12:26.055598] --no-shconf [2022-10-25 23:12:26.055606] -l 0 [2022-10-25 23:12:26.055614] --huge-unlink [2022-10-25 23:12:26.055621] --log-level=lib.eal:6 [2022-10-25 23:12:26.055628] --log-level=lib.cryptodev:5 [2022-10-25 23:12:26.055636] --log-level=user1:6 [2022-10-25 23:12:26.055643] --iova-mode=pa [2022-10-25 23:12:26.055650] --base-virtaddr=0x200000000000 [2022-10-25 23:12:26.055657] --match-allocations [2022-10-25 23:12:26.055665] --file-prefix=spdk_pid83875 [2022-10-25 23:12:26.055672] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:26.194021] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:26.245587] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:26.246263] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_aio_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_aio_foo.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  220435.91 IOPS     861.08 MiB/s

 =============================================================

 Total                       :  220435.91 IOPS     861.08 MiB/s
[2022-10-25 23:12:39.062862] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:39.062933] [ DPDK EAL parameters: [2022-10-25 23:12:39.062944] bdevperf [2022-10-25 23:12:39.062952] --no-shconf [2022-10-25 23:12:39.062961] -l 0 [2022-10-25 23:12:39.062969] --huge-unlink [2022-10-25 23:12:39.062978] --log-level=lib.eal:6 [2022-10-25 23:12:39.062985] --log-level=lib.cryptodev:5 [2022-10-25 23:12:39.062992] --log-level=user1:6 [2022-10-25 23:12:39.063000] --iova-mode=pa [2022-10-25 23:12:39.063008] --base-virtaddr=0x200000000000 [2022-10-25 23:12:39.063016] --match-allocations [2022-10-25 23:12:39.063024] --file-prefix=spdk_pid83901 [2022-10-25 23:12:39.063033] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:39.201528] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:39.258472] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:39.259141] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  218110.54 IOPS     851.99 MiB/s

 =============================================================

 Total                       :  218110.54 IOPS     851.99 MiB/s
[2022-10-25 23:12:52.069012] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:12:52.069081] [ DPDK EAL parameters: [2022-10-25 23:12:52.069091] bdevperf [2022-10-25 23:12:52.069100] --no-shconf [2022-10-25 23:12:52.069108] -l 0 [2022-10-25 23:12:52.069116] --huge-unlink [2022-10-25 23:12:52.069124] --log-level=lib.eal:6 [2022-10-25 23:12:52.069131] --log-level=lib.cryptodev:5 [2022-10-25 23:12:52.069139] --log-level=user1:6 [2022-10-25 23:12:52.069146] --iova-mode=pa [2022-10-25 23:12:52.069153] --base-virtaddr=0x200000000000 [2022-10-25 23:12:52.069161] --match-allocations [2022-10-25 23:12:52.069168] --file-prefix=spdk_pid83926 [2022-10-25 23:12:52.069176] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:12:52.207851] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:12:52.253574] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:12:52.254257] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  217774.07 IOPS     850.68 MiB/s

 =============================================================

 Total                       :  217774.07 IOPS     850.68 MiB/s
[2022-10-25 23:13:05.092769] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:05.092840] [ DPDK EAL parameters: [2022-10-25 23:13:05.092850] bdevperf [2022-10-25 23:13:05.092859] --no-shconf [2022-10-25 23:13:05.092868] -l 0 [2022-10-25 23:13:05.092876] --huge-unlink [2022-10-25 23:13:05.092885] --log-level=lib.eal:6 [2022-10-25 23:13:05.092894] --log-level=lib.cryptodev:5 [2022-10-25 23:13:05.092901] --log-level=user1:6 [2022-10-25 23:13:05.092909] --iova-mode=pa [2022-10-25 23:13:05.092918] --base-virtaddr=0x200000000000 [2022-10-25 23:13:05.092926] --match-allocations [2022-10-25 23:13:05.092934] --file-prefix=spdk_pid83952 [2022-10-25 23:13:05.092943] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:05.231410] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:05.285594] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:05.286255] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_libaio.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_libaio.conf -q 4 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  217800.45 IOPS     850.78 MiB/s

 =============================================================

 Total                       :  217800.45 IOPS     850.78 MiB/s
[2022-10-25 23:13:18.110405] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:18.110473] [ DPDK EAL parameters: [2022-10-25 23:13:18.110484] bdevperf [2022-10-25 23:13:18.110492] --no-shconf [2022-10-25 23:13:18.110500] -l 0 [2022-10-25 23:13:18.110508] --huge-unlink [2022-10-25 23:13:18.110515] --log-level=lib.eal:6 [2022-10-25 23:13:18.110523] --log-level=lib.cryptodev:5 [2022-10-25 23:13:18.110530] --log-level=user1:6 [2022-10-25 23:13:18.110538] --iova-mode=pa [2022-10-25 23:13:18.110544] --base-virtaddr=0x200000000000 [2022-10-25 23:13:18.110552] --match-allocations [2022-10-25 23:13:18.110559] --file-prefix=spdk_pid83977 [2022-10-25 23:13:18.110566] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:18.249238] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:18.293492] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:18.294175] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :       0.06 IOPS       0.00 MiB/s

 =============================================================

 Total                       :       0.06 IOPS       0.00 MiB/s
[2022-10-25 23:13:31.240110] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:13:31.240182] [ DPDK EAL parameters: [2022-10-25 23:13:31.240193] bdevperf [2022-10-25 23:13:31.240202] --no-shconf [2022-10-25 23:13:31.240210] -l 0 [2022-10-25 23:13:31.240219] --huge-unlink [2022-10-25 23:13:31.240227] --log-level=lib.eal:6 [2022-10-25 23:13:31.240234] --log-level=lib.cryptodev:5 [2022-10-25 23:13:31.240241] --log-level=user1:6 [2022-10-25 23:13:31.240249] --iova-mode=pa [2022-10-25 23:13:31.240257] --base-virtaddr=0x200000000000 [2022-10-25 23:13:31.240265] --match-allocations [2022-10-25 23:13:31.240272] --file-prefix=spdk_pid84002 [2022-10-25 23:13:31.240279] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:13:31.378840] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:13:31.433486] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:13:31.434166] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  222460.54 IOPS     868.99 MiB/s

 =============================================================

 Total                       :  222460.54 IOPS     868.99 MiB/s
[2022-10-25 23:20:11.844287] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:11.844351] [ DPDK EAL parameters: [2022-10-25 23:20:11.844362] bdevperf [2022-10-25 23:20:11.844371] --no-shconf [2022-10-25 23:20:11.844380] -l 0 [2022-10-25 23:20:11.844388] --huge-unlink [2022-10-25 23:20:11.844396] --log-level=lib.eal:6 [2022-10-25 23:20:11.844405] --log-level=lib.cryptodev:5 [2022-10-25 23:20:11.844412] --log-level=user1:6 [2022-10-25 23:20:11.844438] --iova-mode=pa [2022-10-25 23:20:11.844448] --base-virtaddr=0x200000000000 [2022-10-25 23:20:11.844456] --match-allocations [2022-10-25 23:20:11.844465] --file-prefix=spdk_pid84034 [2022-10-25 23:20:11.844481] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:11.982803] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:12.037539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:12.038213] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_xnvme_io_uring_cmd.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_xnvme_io_uring_cmd.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  222470.91 IOPS     869.03 MiB/s

 =============================================================

 Total                       :  222470.91 IOPS     869.03 MiB/s
[2022-10-25 23:20:24.779813] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:24.779881] [ DPDK EAL parameters: [2022-10-25 23:20:24.779892] bdevperf [2022-10-25 23:20:24.779901] --no-shconf [2022-10-25 23:20:24.779910] -l 0 [2022-10-25 23:20:24.779918] --huge-unlink [2022-10-25 23:20:24.779927] --log-level=lib.eal:6 [2022-10-25 23:20:24.779934] --log-level=lib.cryptodev:5 [2022-10-25 23:20:24.779942] --log-level=user1:6 [2022-10-25 23:20:24.779950] --iova-mode=pa [2022-10-25 23:20:24.779958] --base-virtaddr=0x200000000000 [2022-10-25 23:20:24.779966] --match-allocations [2022-10-25 23:20:24.779974] --file-prefix=spdk_pid84061 [2022-10-25 23:20:24.779983] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:24.918965] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:24.961539] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:24.962206] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true
$ /root/git/spdk/test/bdev/bdevperf/bdevperf --json /tmp/spdk_bdev_uring_foo.conf -q 8 -o 4096 -w randread -t 10 -m '[0]'
Running I/O for 10 seconds...

 Job: Nvme0n1 (Core Mask 0x1)
	 Nvme0n1             :  223754.18 IOPS     874.04 MiB/s

 =============================================================

 Total                       :  223754.18 IOPS     874.04 MiB/s
[2022-10-25 23:20:37.720662] Starting SPDK v23.01-pre git sha1 fd41a2937 / DPDK 22.07.0 initialization...
[2022-10-25 23:20:37.720732] [ DPDK EAL parameters: [2022-10-25 23:20:37.720742] bdevperf [2022-10-25 23:20:37.720751] --no-shconf [2022-10-25 23:20:37.720760] -l 0 [2022-10-25 23:20:37.720768] --huge-unlink [2022-10-25 23:20:37.720776] --log-level=lib.eal:6 [2022-10-25 23:20:37.720784] --log-level=lib.cryptodev:5 [2022-10-25 23:20:37.720792] --log-level=user1:6 [2022-10-25 23:20:37.720800] --iova-mode=pa [2022-10-25 23:20:37.720809] --base-virtaddr=0x200000000000 [2022-10-25 23:20:37.720817] --match-allocations [2022-10-25 23:20:37.720825] --file-prefix=spdk_pid84086 [2022-10-25 23:20:37.720833] ]
TELEMETRY: No legacy callbacks, legacy socket not created
[2022-10-25 23:20:37.859472] app.c: 586:spdk_app_start: *NOTICE*: Total cores available: 1
[2022-10-25 23:20:37.909520] reactor.c: 926:reactor_run: *NOTICE*: Reactor started on core 0
[2022-10-25 23:20:37.910195] accel_sw.c: 461:sw_accel_module_init: *NOTICE*: Accel framework software module initialized.

$ rm /tmp/spdk_bdev_uring_foo.conf || true

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Run fio...

{'description': '\n    Run fio...\n',
 'extras': {'runlog': {'cmd_01': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.output'),
                                  'state': {'begin': 1666725901.2728775,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7901942729949951,
                                            'end': 1666725902.0630717,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_01.state')},
                       'cmd_02': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.output'),
                                  'state': {'begin': 1666725902.0654507,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.068115472793579,
                                            'end': 1666725913.1335661,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_02.state')},
                       'cmd_03': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733002,\n'
                                            '  "timestamp_ms" : '
                                            '1666733002638,\n'
                                            '  "time" : "Tue Oct 25 23:23:22 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1880924160,\n'
                                            '        "io_kbytes" : 1836840,\n'
                                            '        "bw_bytes" : 268665070,\n'
                                            '        "bw" : 262368,\n'
                                            '        "iops" : 65591.915441,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 459209,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2184,\n'
                                            '          "max" : 28192,\n'
                                            '          "mean" : 2371.134109,\n'
                                            '          "stddev" : 209.422890,\n'
                                            '          "N" : 459210\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 241,\n'
                                            '          "max" : 1211963,\n'
                                            '          "mean" : 12255.930946,\n'
                                            '          "stddev" : '
                                            '2652.769239,\n'
                                            '          "N" : 459209,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11712,\n'
                                            '            "5.000000" : 11712,\n'
                                            '            "10.000000" : 11840,\n'
                                            '            "20.000000" : 11840,\n'
                                            '            "30.000000" : 11840,\n'
                                            '            "40.000000" : 11968,\n'
                                            '            "50.000000" : 11968,\n'
                                            '            "60.000000" : 12096,\n'
                                            '            "70.000000" : 12096,\n'
                                            '            "80.000000" : 12096,\n'
                                            '            "90.000000" : 12224,\n'
                                            '            "95.000000" : 12352,\n'
                                            '            "99.000000" : 21888,\n'
                                            '            "99.500000" : 23424,\n'
                                            '            "99.900000" : 31872,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 64256\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 11632,\n'
                                            '          "max" : 1214328,\n'
                                            '          "mean" : 14627.064896,\n'
                                            '          "stddev" : '
                                            '2660.860079,\n'
                                            '          "N" : 459209\n'
                                            '        },\n'
                                            '        "bw_min" : 260360,\n'
                                            '        "bw_max" : 263680,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '262437.714286,\n'
                                            '        "bw_dev" : 778.330702,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 65090,\n'
                                            '        "iops_max" : 65920,\n'
                                            '        "iops_mean" : '
                                            '65609.428571,\n'
                                            '        "iops_stddev" : '
                                            '194.582675,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.857143,\n'
                                            '      "sys_cpu" : 36.157143,\n'
                                            '      "ctx" : 459183,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.399857,\n'
                                            '        "50" : 1.534595,\n'
                                            '        "100" : 0.054877,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84392: Tue Oct 25 23:23:22 '
                                            '2022\n'
                                            '  read: IOPS=65.6k, BW=256MiB/s '
                                            '(269MB/s)(1794MiB/7001msec)\n'
                                            '    slat (nsec): min=2184, '
                                            'max=28192, avg=2371.13, '
                                            'stdev=209.42\n'
                                            '    clat (nsec): min=241, '
                                            'max=1212.0k, avg=12255.93, '
                                            'stdev=2652.77\n'
                                            '     lat (usec): min=11, '
                                            'max=1214, avg=14.63, stdev= 2.66\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11712],  '
                                            '5.00th=[11712], 10.00th=[11840], '
                                            '20.00th=[11840],\n'
                                            '     | 30.00th=[11840], '
                                            '40.00th=[11968], 50.00th=[11968], '
                                            '60.00th=[12096],\n'
                                            '     | 70.00th=[12096], '
                                            '80.00th=[12096], 90.00th=[12224], '
                                            '95.00th=[12352],\n'
                                            '     | 99.00th=[21888], '
                                            '99.50th=[23424], 99.90th=[31872], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[64256]\n'
                                            '   bw (  KiB/s): min=260360, '
                                            'max=263680, per=100.00%, '
                                            'avg=262437.71, stdev=778.33, '
                                            'samples=14\n'
                                            '   iops        : min=65090, '
                                            'max=65920, avg=65609.43, '
                                            'stdev=194.58, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.40%, 50=1.53%, '
                                            '100=0.05%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  lat (msec)   : 2=0.01%\n'
                                            '  cpu          : usr=9.86%, '
                                            'sys=36.16%, ctx=459183, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=459209,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=256MiB/s (269MB/s), '
                                            '256MiB/s-256MiB/s '
                                            '(269MB/s-269MB/s), io=1794MiB '
                                            '(1881MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.output'),
                                  'state': {'begin': 1666725913.1364834,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7891125679016113,
                                            'end': 1666725913.925596,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_03.state')},
                       'cmd_04': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.output'),
                                  'state': {'begin': 1666725914.9139788,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7808358669281006,
                                            'end': 1666725915.6948147,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_04.state')},
                       'cmd_05': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.output'),
                                  'state': {'begin': 1666725915.696932,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.044736385345459,
                                            'end': 1666725926.7416685,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_05.state')},
                       'cmd_06': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733016,\n'
                                            '  "timestamp_ms" : '
                                            '1666733016245,\n'
                                            '  "time" : "Tue Oct 25 23:23:36 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1887145984,\n'
                                            '        "io_kbytes" : 1842916,\n'
                                            '        "bw_bytes" : 269553775,\n'
                                            '        "bw" : 263236,\n'
                                            '        "iops" : 65809.027282,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 460729,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2174,\n'
                                            '          "max" : 31970,\n'
                                            '          "mean" : 2353.754324,\n'
                                            '          "stddev" : 207.012099,\n'
                                            '          "N" : 460729\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 200,\n'
                                            '          "max" : 183274,\n'
                                            '          "mean" : 12205.161577,\n'
                                            '          "stddev" : '
                                            '1978.440122,\n'
                                            '          "N" : 460729,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11712,\n'
                                            '            "5.000000" : 11712,\n'
                                            '            "10.000000" : 11712,\n'
                                            '            "20.000000" : 11840,\n'
                                            '            "30.000000" : 11840,\n'
                                            '            "40.000000" : 11840,\n'
                                            '            "50.000000" : 11968,\n'
                                            '            "60.000000" : 11968,\n'
                                            '            "70.000000" : 12096,\n'
                                            '            "80.000000" : 12096,\n'
                                            '            "90.000000" : 12096,\n'
                                            '            "95.000000" : 12352,\n'
                                            '            "99.000000" : 21888,\n'
                                            '            "99.500000" : 23168,\n'
                                            '            "99.900000" : 31360,\n'
                                            '            "99.950000" : 49920,\n'
                                            '            "99.990000" : 64256\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 11591,\n'
                                            '          "max" : 185708,\n'
                                            '          "mean" : 14558.915901,\n'
                                            '          "stddev" : '
                                            '1988.143930,\n'
                                            '          "N" : 460729\n'
                                            '        },\n'
                                            '        "bw_min" : 262208,\n'
                                            '        "bw_max" : 263920,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '263299.928571,\n'
                                            '        "bw_dev" : 429.320118,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 65552,\n'
                                            '        "iops_max" : 65980,\n'
                                            '        "iops_mean" : '
                                            '65824.928571,\n'
                                            '        "iops_stddev" : '
                                            '107.268509,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.728571,\n'
                                            '      "sys_cpu" : 36.385714,\n'
                                            '      "ctx" : 460697,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 1,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.392330,\n'
                                            '        "50" : 1.542989,\n'
                                            '        "100" : 0.051874,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84430: Tue Oct 25 23:23:36 '
                                            '2022\n'
                                            '  read: IOPS=65.8k, BW=257MiB/s '
                                            '(270MB/s)(1800MiB/7001msec)\n'
                                            '    slat (nsec): min=2174, '
                                            'max=31970, avg=2353.75, '
                                            'stdev=207.01\n'
                                            '    clat (nsec): min=200, '
                                            'max=183274, avg=12205.16, '
                                            'stdev=1978.44\n'
                                            '     lat (usec): min=11, max=185, '
                                            'avg=14.56, stdev= 1.99\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11712],  '
                                            '5.00th=[11712], 10.00th=[11712], '
                                            '20.00th=[11840],\n'
                                            '     | 30.00th=[11840], '
                                            '40.00th=[11840], 50.00th=[11968], '
                                            '60.00th=[11968],\n'
                                            '     | 70.00th=[12096], '
                                            '80.00th=[12096], 90.00th=[12096], '
                                            '95.00th=[12352],\n'
                                            '     | 99.00th=[21888], '
                                            '99.50th=[23168], 99.90th=[31360], '
                                            '99.95th=[49920],\n'
                                            '     | 99.99th=[64256]\n'
                                            '   bw (  KiB/s): min=262208, '
                                            'max=263920, per=100.00%, '
                                            'avg=263299.93, stdev=429.32, '
                                            'samples=14\n'
                                            '   iops        : min=65552, '
                                            'max=65980, avg=65824.93, '
                                            'stdev=107.27, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.39%, 50=1.54%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=9.73%, '
                                            'sys=36.39%, ctx=460697, majf=0, '
                                            'minf=1\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=460729,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=257MiB/s (270MB/s), '
                                            '257MiB/s-257MiB/s '
                                            '(270MB/s-270MB/s), io=1800MiB '
                                            '(1887MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.output'),
                                  'state': {'begin': 1666725926.7443569,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7978301048278809,
                                            'end': 1666725927.542187,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_06.state')},
                       'cmd_07': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.output'),
                                  'state': {'begin': 1666725928.5928621,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7860045433044434,
                                            'end': 1666725929.3788667,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_07.state')},
                       'cmd_08': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.output'),
                                  'state': {'begin': 1666725929.3810534,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring_cmd" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.049619674682617,
                                            'end': 1666725940.4306731,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_08.state')},
                       'cmd_09': {'output': 'io_uring_cmd-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring_cmd, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733029,\n'
                                            '  "timestamp_ms" : '
                                            '1666733029933,\n'
                                            '  "time" : "Tue Oct 25 23:23:49 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"io_uring_cmd",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1777991680,\n'
                                            '        "io_kbytes" : 1736320,\n'
                                            '        "bw_bytes" : 253962531,\n'
                                            '        "bw" : 248010,\n'
                                            '        "iops" : 62002.571061,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 434080,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2184,\n'
                                            '          "max" : 31269,\n'
                                            '          "mean" : 2476.576864,\n'
                                            '          "stddev" : 207.606246,\n'
                                            '          "N" : 434080\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 250,\n'
                                            '          "max" : 133119,\n'
                                            '          "mean" : 13071.487523,\n'
                                            '          "stddev" : '
                                            '2165.425154,\n'
                                            '          "N" : 434080,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 11968,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12096,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 13248,\n'
                                            '            "60.000000" : 13376,\n'
                                            '            "70.000000" : 13376,\n'
                                            '            "80.000000" : 13376,\n'
                                            '            "90.000000" : 13504,\n'
                                            '            "95.000000" : 13632,\n'
                                            '            "99.000000" : 22912,\n'
                                            '            "99.500000" : 29312,\n'
                                            '            "99.900000" : 32640,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 68096\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 13816,\n'
                                            '          "max" : 135424,\n'
                                            '          "mean" : 15548.064387,\n'
                                            '          "stddev" : '
                                            '2206.175957,\n'
                                            '          "N" : 434080\n'
                                            '        },\n'
                                            '        "bw_min" : 237664,\n'
                                            '        "bw_max" : 261000,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '248095.357143,\n'
                                            '        "bw_dev" : 11422.448246,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 59416,\n'
                                            '        "iops_max" : 65250,\n'
                                            '        "iops_mean" : '
                                            '62023.785714,\n'
                                            '        "iops_stddev" : '
                                            '2855.663204,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.014286,\n'
                                            '      "sys_cpu" : 35.057143,\n'
                                            '      "ctx" : 434068,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.284187,\n'
                                            '        "50" : 1.652691,\n'
                                            '        "100" : 0.052525,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-reference: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84468: Tue Oct 25 23:23:49 '
                                            '2022\n'
                                            '  read: IOPS=62.0k, BW=242MiB/s '
                                            '(254MB/s)(1696MiB/7001msec)\n'
                                            '    slat (nsec): min=2184, '
                                            'max=31269, avg=2476.58, '
                                            'stdev=207.61\n'
                                            '    clat (nsec): min=250, '
                                            'max=133119, avg=13071.49, '
                                            'stdev=2165.43\n'
                                            '     lat (usec): min=13, max=135, '
                                            'avg=15.55, stdev= 2.21\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[11968], 10.00th=[12096], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12096], '
                                            '40.00th=[12224], 50.00th=[13248], '
                                            '60.00th=[13376],\n'
                                            '     | 70.00th=[13376], '
                                            '80.00th=[13376], 90.00th=[13504], '
                                            '95.00th=[13632],\n'
                                            '     | 99.00th=[22912], '
                                            '99.50th=[29312], 99.90th=[32640], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[68096]\n'
                                            '   bw (  KiB/s): min=237664, '
                                            'max=261000, per=100.00%, '
                                            'avg=248095.36, stdev=11422.45, '
                                            'samples=14\n'
                                            '   iops        : min=59416, '
                                            'max=65250, avg=62023.79, '
                                            'stdev=2855.66, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.28%, 50=1.65%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=8.01%, '
                                            'sys=35.06%, ctx=434068, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=434080,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=242MiB/s (254MB/s), '
                                            '242MiB/s-242MiB/s '
                                            '(254MB/s-254MB/s), io=1696MiB '
                                            '(1778MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.output'),
                                  'state': {'begin': 1666725940.4336092,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7855269908905029,
                                            'end': 1666725941.2191362,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_09.state')},
                       'cmd_10': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.output'),
                                  'state': {'begin': 1666725942.249782,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7795379161834717,
                                            'end': 1666725943.02932,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_10.state')},
                       'cmd_100': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.output'),
                                   'state': {'begin': 1666726373.990537,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7794060707092285,
                                             'end': 1666726374.769943,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_100.state')},
                       'cmd_101': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.output'),
                                   'state': {'begin': 1666726374.7717996,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.049149751663208,
                                             'end': 1666726385.8209493,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_101.state')},
                       'cmd_102': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733475,\n'
                                             '  "timestamp_ms" : '
                                             '1666733475316,\n'
                                             '  "time" : "Tue Oct 25 23:31:15 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2836426752,\n'
                                             '        "io_kbytes" : 2769948,\n'
                                             '        "bw_bytes" : 405145943,\n'
                                             '        "bw" : 395650,\n'
                                             '        "iops" : 98912.298243,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 692485,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1893,\n'
                                             '          "max" : 27962,\n'
                                             '          "mean" : 2200.003763,\n'
                                             '          "stddev" : '
                                             '323.295581,\n'
                                             '          "N" : 692486\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3106,\n'
                                             '          "max" : 189856,\n'
                                             '          "mean" : '
                                             '17485.482183,\n'
                                             '          "stddev" : '
                                             '3150.118639,\n'
                                             '          "N" : 692486,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16320,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16512,\n'
                                             '            "30.000000" : '
                                             '16512,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '17024,\n'
                                             '            "60.000000" : '
                                             '17024,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17280,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '33536,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '62720,\n'
                                             '            "99.990000" : 73216\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11261,\n'
                                             '          "max" : 191810,\n'
                                             '          "mean" : '
                                             '19685.485946,\n'
                                             '          "stddev" : '
                                             '3142.083143,\n'
                                             '          "N" : 692486\n'
                                             '        },\n'
                                             '        "bw_min" : 394512,\n'
                                             '        "bw_max" : 396576,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '395746.285714,\n'
                                             '        "bw_dev" : 618.185363,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 98628,\n'
                                             '        "iops_max" : 99144,\n'
                                             '        "iops_mean" : '
                                             '98936.571429,\n'
                                             '        "iops_stddev" : '
                                             '154.546341,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.085714,\n'
                                             '      "sys_cpu" : 38.200000,\n'
                                             '      "ctx" : 346121,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010686,\n'
                                             '        "20" : 95.026318,\n'
                                             '        "50" : 4.855268,\n'
                                             '        "100" : 0.104840,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=85837: Tue Oct 25 23:31:15 '
                                             '2022\n'
                                             '  read: IOPS=98.9k, BW=386MiB/s '
                                             '(405MB/s)(2705MiB/7001msec)\n'
                                             '    slat (nsec): min=1893, '
                                             'max=27962, avg=2200.00, '
                                             'stdev=323.30\n'
                                             '    clat (usec): min=3, max=189, '
                                             'avg=17.49, stdev= 3.15\n'
                                             '     lat (usec): min=11, '
                                             'max=191, avg=19.69, stdev= 3.14\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16320],  '
                                             '5.00th=[16512], 10.00th=[16512], '
                                             '20.00th=[16512],\n'
                                             '     | 30.00th=[16512], '
                                             '40.00th=[16768], '
                                             '50.00th=[17024], '
                                             '60.00th=[17024],\n'
                                             '     | 70.00th=[17024], '
                                             '80.00th=[17280], '
                                             '90.00th=[17280], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[33536], '
                                             '99.50th=[34560], '
                                             '99.90th=[50432], '
                                             '99.95th=[62720],\n'
                                             '     | 99.99th=[73216]\n'
                                             '   bw (  KiB/s): min=394512, '
                                             'max=396576, per=100.00%, '
                                             'avg=395746.29, stdev=618.19, '
                                             'samples=14\n'
                                             '   iops        : min=98628, '
                                             'max=99144, avg=98936.57, '
                                             'stdev=154.55, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=95.03%, 50=4.86%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=12.09%, '
                                             'sys=38.20%, ctx=346121, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=692485,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=386MiB/s (405MB/s), '
                                             '386MiB/s-386MiB/s '
                                             '(405MB/s-405MB/s), io=2705MiB '
                                             '(2836MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.output'),
                                   'state': {'begin': 1666726385.8235474,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7883989810943604,
                                             'end': 1666726386.6119463,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_102.state')},
                       'cmd_103': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.output'),
                                   'state': {'begin': 1666726387.6444223,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7858450412750244,
                                             'end': 1666726388.4302673,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_103.state')},
                       'cmd_104': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.output'),
                                   'state': {'begin': 1666726388.4321644,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.038004875183105,
                                             'end': 1666726399.4701693,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_104.state')},
                       'cmd_105': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733488,\n'
                                             '  "timestamp_ms" : '
                                             '1666733488966,\n'
                                             '  "time" : "Tue Oct 25 23:31:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2790170624,\n'
                                             '        "io_kbytes" : 2724776,\n'
                                             '        "bw_bytes" : 398538869,\n'
                                             '        "bw" : 389198,\n'
                                             '        "iops" : 97299.242965,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 681192,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1963,\n'
                                             '          "max" : 37029,\n'
                                             '          "mean" : 2279.325263,\n'
                                             '          "stddev" : '
                                             '339.765872,\n'
                                             '          "N" : 681193\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3075,\n'
                                             '          "max" : 121388,\n'
                                             '          "mean" : '
                                             '17720.137531,\n'
                                             '          "stddev" : '
                                             '3211.933311,\n'
                                             '          "N" : 681193,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17280,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '20096,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '35072,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63232,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11381,\n'
                                             '          "max" : 123462,\n'
                                             '          "mean" : '
                                             '19999.462794,\n'
                                             '          "stddev" : '
                                             '3203.796715,\n'
                                             '          "N" : 681193\n'
                                             '        },\n'
                                             '        "bw_min" : 388424,\n'
                                             '        "bw_max" : 390448,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '389292.714286,\n'
                                             '        "bw_dev" : 718.143593,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 97106,\n'
                                             '        "iops_max" : 97612,\n'
                                             '        "iops_mean" : '
                                             '97323.142857,\n'
                                             '        "iops_stddev" : '
                                             '179.542520,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 11.500000,\n'
                                             '      "sys_cpu" : 39.285714,\n'
                                             '      "ctx" : 340448,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.011304,\n'
                                             '        "20" : 94.985261,\n'
                                             '        "50" : 4.893921,\n'
                                             '        "100" : 0.104082,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=85875: Tue Oct 25 23:31:28 '
                                             '2022\n'
                                             '  read: IOPS=97.3k, BW=380MiB/s '
                                             '(399MB/s)(2661MiB/7001msec)\n'
                                             '    slat (nsec): min=1963, '
                                             'max=37029, avg=2279.33, '
                                             'stdev=339.77\n'
                                             '    clat (usec): min=3, max=121, '
                                             'avg=17.72, stdev= 3.21\n'
                                             '     lat (usec): min=11, '
                                             'max=123, avg=20.00, stdev= 3.20\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16512],  '
                                             '5.00th=[16512], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[17280], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17280], '
                                             '80.00th=[17536], '
                                             '90.00th=[17536], '
                                             '95.00th=[20096],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[35072], '
                                             '99.90th=[50432], '
                                             '99.95th=[63232],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=388424, '
                                             'max=390448, per=100.00%, '
                                             'avg=389292.71, stdev=718.14, '
                                             'samples=14\n'
                                             '   iops        : min=97106, '
                                             'max=97612, avg=97323.14, '
                                             'stdev=179.54, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=94.99%, 50=4.89%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=11.50%, '
                                             'sys=39.29%, ctx=340448, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=681192,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=380MiB/s (399MB/s), '
                                             '380MiB/s-380MiB/s '
                                             '(399MB/s-399MB/s), io=2661MiB '
                                             '(2790MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.output'),
                                   'state': {'begin': 1666726399.4770014,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7884995937347412,
                                             'end': 1666726400.265501,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_105.state')},
                       'cmd_106': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.output'),
                                   'state': {'begin': 1666726401.2915058,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7873985767364502,
                                             'end': 1666726402.0789044,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_106.state')},
                       'cmd_107': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.output'),
                                   'state': {'begin': 1666726402.0813653,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.045714855194092,
                                             'end': 1666726413.1270802,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_107.state')},
                       'cmd_108': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733502,\n'
                                             '  "timestamp_ms" : '
                                             '1666733502622,\n'
                                             '  "time" : "Tue Oct 25 23:31:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2812887040,\n'
                                             '        "io_kbytes" : 2746960,\n'
                                             '        "bw_bytes" : 401783608,\n'
                                             '        "bw" : 392366,\n'
                                             '        "iops" : 98091.558349,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 686739,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1884,\n'
                                             '          "max" : 31248,\n'
                                             '          "mean" : 2222.947482,\n'
                                             '          "stddev" : '
                                             '326.104416,\n'
                                             '          "N" : 686739\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2996,\n'
                                             '          "max" : 255970,\n'
                                             '          "mean" : '
                                             '17635.028845,\n'
                                             '          "stddev" : '
                                             '3256.648995,\n'
                                             '          "N" : 686740,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '17024,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17280,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '20096,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11231,\n'
                                             '          "max" : 257904,\n'
                                             '          "mean" : '
                                             '19857.976051,\n'
                                             '          "stddev" : '
                                             '3249.803901,\n'
                                             '          "N" : 686740\n'
                                             '        },\n'
                                             '        "bw_min" : 388705,\n'
                                             '        "bw_max" : 395896,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '392460.642857,\n'
                                             '        "bw_dev" : 2587.148404,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 97176,\n'
                                             '        "iops_max" : 98974,\n'
                                             '        "iops_mean" : '
                                             '98115.142857,\n'
                                             '        "iops_stddev" : '
                                             '646.815020,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.271429,\n'
                                             '      "sys_cpu" : 38.271429,\n'
                                             '      "ctx" : 343225,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.014416,\n'
                                             '        "20" : 94.981354,\n'
                                             '        "50" : 4.886718,\n'
                                             '        "100" : 0.113435,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=85913: Tue Oct 25 23:31:42 '
                                             '2022\n'
                                             '  read: IOPS=98.1k, BW=383MiB/s '
                                             '(402MB/s)(2683MiB/7001msec)\n'
                                             '    slat (nsec): min=1884, '
                                             'max=31248, avg=2222.95, '
                                             'stdev=326.10\n'
                                             '    clat (usec): min=2, max=255, '
                                             'avg=17.64, stdev= 3.26\n'
                                             '     lat (usec): min=11, '
                                             'max=257, avg=19.86, stdev= 3.25\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16512],  '
                                             '5.00th=[16512], 10.00th=[16512], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[17024], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17280], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[20096],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[34560], '
                                             '99.90th=[50944], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=388705, '
                                             'max=395896, per=100.00%, '
                                             'avg=392460.64, stdev=2587.15, '
                                             'samples=14\n'
                                             '   iops        : min=97176, '
                                             'max=98974, avg=98115.14, '
                                             'stdev=646.82, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=94.98%, 50=4.89%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  cpu          : usr=12.27%, '
                                             'sys=38.27%, ctx=343225, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=686739,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=383MiB/s (402MB/s), '
                                             '383MiB/s-383MiB/s '
                                             '(402MB/s-402MB/s), io=2683MiB '
                                             '(2813MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.output'),
                                   'state': {'begin': 1666726413.1298363,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.788353443145752,
                                             'end': 1666726413.9181898,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_108.state')},
                       'cmd_109': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.output'),
                                   'state': {'begin': 1666726414.953366,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7746560573577881,
                                             'end': 1666726415.728022,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_109.state')},
                       'cmd_11': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.output'),
                                  'state': {'begin': 1666725943.0313234,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.058010816574097,
                                            'end': 1666725954.0893342,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_11.state')},
                       'cmd_110': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.output'),
                                   'state': {'begin': 1666726415.7300074,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.052646398544312,
                                             'end': 1666726426.7826538,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_110.state')},
                       'cmd_111': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733516,\n'
                                             '  "timestamp_ms" : '
                                             '1666733516277,\n'
                                             '  "time" : "Tue Oct 25 23:31:56 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2816339968,\n'
                                             '        "io_kbytes" : 2750332,\n'
                                             '        "bw_bytes" : 402276813,\n'
                                             '        "bw" : 392848,\n'
                                             '        "iops" : 98211.969719,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 687582,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2003,\n'
                                             '          "max" : 40265,\n'
                                             '          "mean" : 2351.189732,\n'
                                             '          "stddev" : '
                                             '337.786924,\n'
                                             '          "N" : 687582\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3356,\n'
                                             '          "max" : 209222,\n'
                                             '          "mean" : '
                                             '17502.922175,\n'
                                             '          "stddev" : '
                                             '3232.779953,\n'
                                             '          "N" : 687583,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16320,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16512,\n'
                                             '            "30.000000" : '
                                             '16512,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '17024,\n'
                                             '            "60.000000" : '
                                             '17024,\n'
                                             '            "70.000000" : '
                                             '17280,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17280,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '33536,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11251,\n'
                                             '          "max" : 211306,\n'
                                             '          "mean" : '
                                             '19854.111460,\n'
                                             '          "stddev" : '
                                             '3223.695868,\n'
                                             '          "N" : 687583\n'
                                             '        },\n'
                                             '        "bw_min" : 391559,\n'
                                             '        "bw_max" : 394072,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '392940.500000,\n'
                                             '        "bw_dev" : 744.587315,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 97889,\n'
                                             '        "iops_max" : 98520,\n'
                                             '        "iops_mean" : '
                                             '98235.214286,\n'
                                             '        "iops_stddev" : '
                                             '186.488267,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.885714,\n'
                                             '      "sys_cpu" : 39.400000,\n'
                                             '      "ctx" : 343653,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010471,\n'
                                             '        "20" : 95.019649,\n'
                                             '        "50" : 4.858766,\n'
                                             '        "100" : 0.108642,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=85953: Tue '
                                             'Oct 25 23:31:56 2022\n'
                                             '  read: IOPS=98.2k, BW=384MiB/s '
                                             '(402MB/s)(2686MiB/7001msec)\n'
                                             '    slat (nsec): min=2003, '
                                             'max=40265, avg=2351.19, '
                                             'stdev=337.79\n'
                                             '    clat (usec): min=3, max=209, '
                                             'avg=17.50, stdev= 3.23\n'
                                             '     lat (usec): min=11, '
                                             'max=211, avg=19.85, stdev= 3.22\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16320],  '
                                             '5.00th=[16512], 10.00th=[16512], '
                                             '20.00th=[16512],\n'
                                             '     | 30.00th=[16512], '
                                             '40.00th=[16768], '
                                             '50.00th=[17024], '
                                             '60.00th=[17024],\n'
                                             '     | 70.00th=[17280], '
                                             '80.00th=[17280], '
                                             '90.00th=[17280], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[33536], '
                                             '99.50th=[34560], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=391559, '
                                             'max=394072, per=100.00%, '
                                             'avg=392940.50, stdev=744.59, '
                                             'samples=14\n'
                                             '   iops        : min=97889, '
                                             'max=98520, avg=98235.21, '
                                             'stdev=186.49, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=95.02%, 50=4.86%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=12.89%, '
                                             'sys=39.40%, ctx=343653, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=687582,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=384MiB/s (402MB/s), '
                                             '384MiB/s-384MiB/s '
                                             '(402MB/s-402MB/s), io=2686MiB '
                                             '(2816MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.output'),
                                   'state': {'begin': 1666726426.7856476,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7886650562286377,
                                             'end': 1666726427.5743127,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_111.state')},
                       'cmd_112': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.output'),
                                   'state': {'begin': 1666726428.5737777,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7817878723144531,
                                             'end': 1666726429.3555655,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_112.state')},
                       'cmd_113': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.output'),
                                   'state': {'begin': 1666726429.35767,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.057876348495483,
                                             'end': 1666726440.4155464,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_113.state')},
                       'cmd_114': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733529,\n'
                                             '  "timestamp_ms" : '
                                             '1666733529910,\n'
                                             '  "time" : "Tue Oct 25 23:32:09 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2630627328,\n'
                                             '        "io_kbytes" : 2568972,\n'
                                             '        "bw_bytes" : 375750225,\n'
                                             '        "bw" : 366943,\n'
                                             '        "iops" : 91735.752035,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 642242,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2124,\n'
                                             '          "max" : 14818,\n'
                                             '          "mean" : 2496.051688,\n'
                                             '          "stddev" : '
                                             '247.656662,\n'
                                             '          "N" : 642242\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4859,\n'
                                             '          "max" : 136816,\n'
                                             '          "mean" : '
                                             '18815.614652,\n'
                                             '          "stddev" : '
                                             '3328.463438,\n'
                                             '          "N" : 642243,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 17536,\n'
                                             '            "5.000000" : 17792,\n'
                                             '            "10.000000" : '
                                             '17792,\n'
                                             '            "20.000000" : '
                                             '17792,\n'
                                             '            "30.000000" : '
                                             '17792,\n'
                                             '            "40.000000" : '
                                             '18048,\n'
                                             '            "50.000000" : '
                                             '18304,\n'
                                             '            "60.000000" : '
                                             '18304,\n'
                                             '            "70.000000" : '
                                             '18304,\n'
                                             '            "80.000000" : '
                                             '18560,\n'
                                             '            "90.000000" : '
                                             '18560,\n'
                                             '            "95.000000" : '
                                             '21888,\n'
                                             '            "99.000000" : '
                                             '35072,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '49920,\n'
                                             '            "99.950000" : '
                                             '62208,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12043,\n'
                                             '          "max" : 139101,\n'
                                             '          "mean" : '
                                             '21311.665947,\n'
                                             '          "stddev" : '
                                             '3318.507739,\n'
                                             '          "N" : 642243\n'
                                             '        },\n'
                                             '        "bw_min" : 366640,\n'
                                             '        "bw_max" : 368008,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '367114.357143,\n'
                                             '        "bw_dev" : 409.196136,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 91660,\n'
                                             '        "iops_max" : 92002,\n'
                                             '        "iops_mean" : '
                                             '91778.500000,\n'
                                             '        "iops_stddev" : '
                                             '102.304034,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.457143,\n'
                                             '      "sys_cpu" : 36.600000,\n'
                                             '      "ctx" : 321062,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 94.633643,\n'
                                             '        "50" : 5.261257,\n'
                                             '        "100" : 0.097627,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=85992: Tue '
                                             'Oct 25 23:32:09 2022\n'
                                             '  read: IOPS=91.7k, BW=358MiB/s '
                                             '(376MB/s)(2509MiB/7001msec)\n'
                                             '    slat (nsec): min=2124, '
                                             'max=14818, avg=2496.05, '
                                             'stdev=247.66\n'
                                             '    clat (usec): min=4, max=136, '
                                             'avg=18.82, stdev= 3.33\n'
                                             '     lat (usec): min=12, '
                                             'max=139, avg=21.31, stdev= 3.32\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[17536],  '
                                             '5.00th=[17792], 10.00th=[17792], '
                                             '20.00th=[17792],\n'
                                             '     | 30.00th=[17792], '
                                             '40.00th=[18048], '
                                             '50.00th=[18304], '
                                             '60.00th=[18304],\n'
                                             '     | 70.00th=[18304], '
                                             '80.00th=[18560], '
                                             '90.00th=[18560], '
                                             '95.00th=[21888],\n'
                                             '     | 99.00th=[35072], '
                                             '99.50th=[44288], '
                                             '99.90th=[49920], '
                                             '99.95th=[62208],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=366640, '
                                             'max=368008, per=100.00%, '
                                             'avg=367114.36, stdev=409.20, '
                                             'samples=14\n'
                                             '   iops        : min=91660, '
                                             'max=92002, avg=91778.50, '
                                             'stdev=102.30, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=94.63%, 50=5.26%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=12.46%, '
                                             'sys=36.60%, ctx=321062, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=642242,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=358MiB/s (376MB/s), '
                                             '358MiB/s-358MiB/s '
                                             '(376MB/s-376MB/s), io=2509MiB '
                                             '(2631MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.output'),
                                   'state': {'begin': 1666726440.4181273,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7908730506896973,
                                             'end': 1666726441.2090003,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_114.state')},
                       'cmd_115': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.output'),
                                   'state': {'begin': 1666726442.2077224,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.782257080078125,
                                             'end': 1666726442.9899795,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_115.state')},
                       'cmd_116': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.output'),
                                   'state': {'begin': 1666726442.992052,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.055752992630005,
                                             'end': 1666726454.047805,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_116.state')},
                       'cmd_117': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733543,\n'
                                             '  "timestamp_ms" : '
                                             '1666733543542,\n'
                                             '  "time" : "Tue Oct 25 23:32:23 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2749571072,\n'
                                             '        "io_kbytes" : 2685128,\n'
                                             '        "bw_bytes" : 392739761,\n'
                                             '        "bw" : 383534,\n'
                                             '        "iops" : 95883.588059,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 671281,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2064,\n'
                                             '          "max" : 31810,\n'
                                             '          "mean" : 2400.967224,\n'
                                             '          "stddev" : '
                                             '306.105301,\n'
                                             '          "N" : 671281\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 391,\n'
                                             '          "max" : 135253,\n'
                                             '          "mean" : '
                                             '17952.067264,\n'
                                             '          "stddev" : '
                                             '3287.475107,\n'
                                             '          "N" : 671282,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '17280,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17792,\n'
                                             '            "80.000000" : '
                                             '18048,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '20352,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11592,\n'
                                             '          "max" : 143409,\n'
                                             '          "mean" : '
                                             '20353.034373,\n'
                                             '          "stddev" : '
                                             '3289.918264,\n'
                                             '          "N" : 671282\n'
                                             '        },\n'
                                             '        "bw_min" : 367224,\n'
                                             '        "bw_max" : 390992,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '383631.928571,\n'
                                             '        "bw_dev" : 9319.619474,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 91806,\n'
                                             '        "iops_max" : 97748,\n'
                                             '        "iops_mean" : '
                                             '95907.928571,\n'
                                             '        "iops_stddev" : '
                                             '2330.003977,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.528571,\n'
                                             '      "sys_cpu" : 37.685714,\n'
                                             '      "ctx" : 335517,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.011471,\n'
                                             '        "20" : 94.923586,\n'
                                             '        "50" : 4.958132,\n'
                                             '        "100" : 0.104278,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86031: Tue '
                                             'Oct 25 23:32:23 2022\n'
                                             '  read: IOPS=95.9k, BW=375MiB/s '
                                             '(393MB/s)(2622MiB/7001msec)\n'
                                             '    slat (nsec): min=2064, '
                                             'max=31810, avg=2400.97, '
                                             'stdev=306.11\n'
                                             '    clat (nsec): min=391, '
                                             'max=135253, avg=17952.07, '
                                             'stdev=3287.48\n'
                                             '     lat (usec): min=11, '
                                             'max=143, avg=20.35, stdev= 3.29\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16512],  '
                                             '5.00th=[16512], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[17280], '
                                             '50.00th=[17280], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17792], '
                                             '80.00th=[18048], '
                                             '90.00th=[18304], '
                                             '95.00th=[20352],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[37120], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=367224, '
                                             'max=390992, per=100.00%, '
                                             'avg=383631.93, stdev=9319.62, '
                                             'samples=14\n'
                                             '   iops        : min=91806, '
                                             'max=97748, avg=95907.93, '
                                             'stdev=2330.00, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=94.92%, 50=4.96%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.53%, '
                                             'sys=37.69%, ctx=335517, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=671281,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=375MiB/s (393MB/s), '
                                             '375MiB/s-375MiB/s '
                                             '(393MB/s-393MB/s), io=2622MiB '
                                             '(2750MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.output'),
                                   'state': {'begin': 1666726454.0507383,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.793297290802002,
                                             'end': 1666726454.8440356,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_117.state')},
                       'cmd_118': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.output'),
                                   'state': {'begin': 1666726456.863824,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7858519554138184,
                                             'end': 1666726457.6496758,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_118.state')},
                       'cmd_119': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.output'),
                                   'state': {'begin': 1666726457.6520848,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.47193956375122,
                                             'end': 1666726469.1240244,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_119.state')},
                       'cmd_12': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733043,\n'
                                            '  "timestamp_ms" : '
                                            '1666733043592,\n'
                                            '  "time" : "Tue Oct 25 23:24:03 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1862557696,\n'
                                            '        "io_kbytes" : 1818904,\n'
                                            '        "bw_bytes" : 266041664,\n'
                                            '        "bw" : 259806,\n'
                                            '        "iops" : 64951.578346,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 454726,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2314,\n'
                                            '          "max" : 40516,\n'
                                            '          "mean" : 2487.807033,\n'
                                            '          "stddev" : 208.501404,\n'
                                            '          "N" : 454726\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 351,\n'
                                            '          "max" : 1341115,\n'
                                            '          "mean" : 12323.750791,\n'
                                            '          "stddev" : '
                                            '2811.725511,\n'
                                            '          "N" : 454726,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11840,\n'
                                            '            "5.000000" : 11840,\n'
                                            '            "10.000000" : 11840,\n'
                                            '            "20.000000" : 11840,\n'
                                            '            "30.000000" : 11968,\n'
                                            '            "40.000000" : 11968,\n'
                                            '            "50.000000" : 12096,\n'
                                            '            "60.000000" : 12096,\n'
                                            '            "70.000000" : 12096,\n'
                                            '            "80.000000" : 12224,\n'
                                            '            "90.000000" : 12224,\n'
                                            '            "95.000000" : 12480,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 23680,\n'
                                            '            "99.900000" : 31616,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 64256\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12694,\n'
                                            '          "max" : 1343630,\n'
                                            '          "mean" : 14811.557824,\n'
                                            '          "stddev" : '
                                            '2820.004980,\n'
                                            '          "N" : 454726\n'
                                            '        },\n'
                                            '        "bw_min" : 259104,\n'
                                            '        "bw_max" : 260440,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '259877.071429,\n'
                                            '        "bw_dev" : 357.766676,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 64776,\n'
                                            '        "iops_max" : 65110,\n'
                                            '        "iops_mean" : '
                                            '64969.214286,\n'
                                            '        "iops_stddev" : '
                                            '89.484144,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.557143,\n'
                                            '      "sys_cpu" : 35.457143,\n'
                                            '      "ctx" : 454702,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.417728,\n'
                                            '        "50" : 1.522235,\n'
                                            '        "100" : 0.051679,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=84507: Tue '
                                            'Oct 25 23:24:03 2022\n'
                                            '  read: IOPS=65.0k, BW=254MiB/s '
                                            '(266MB/s)(1776MiB/7001msec)\n'
                                            '    slat (nsec): min=2314, '
                                            'max=40516, avg=2487.81, '
                                            'stdev=208.50\n'
                                            '    clat (nsec): min=351, '
                                            'max=1341.1k, avg=12323.75, '
                                            'stdev=2811.73\n'
                                            '     lat (usec): min=12, '
                                            'max=1343, avg=14.81, stdev= 2.82\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11840],  '
                                            '5.00th=[11840], 10.00th=[11840], '
                                            '20.00th=[11840],\n'
                                            '     | 30.00th=[11968], '
                                            '40.00th=[11968], 50.00th=[12096], '
                                            '60.00th=[12096],\n'
                                            '     | 70.00th=[12096], '
                                            '80.00th=[12224], 90.00th=[12224], '
                                            '95.00th=[12480],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[23680], 99.90th=[31616], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[64256]\n'
                                            '   bw (  KiB/s): min=259104, '
                                            'max=260440, per=100.00%, '
                                            'avg=259877.07, stdev=357.77, '
                                            'samples=14\n'
                                            '   iops        : min=64776, '
                                            'max=65110, avg=64969.21, '
                                            'stdev=89.48, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.42%, 50=1.52%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  lat (msec)   : 2=0.01%\n'
                                            '  cpu          : usr=11.56%, '
                                            'sys=35.46%, ctx=454702, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=454726,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=254MiB/s (266MB/s), '
                                            '254MiB/s-254MiB/s '
                                            '(266MB/s-266MB/s), io=1776MiB '
                                            '(1863MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.output'),
                                  'state': {'begin': 1666725954.0924244,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.803476095199585,
                                            'end': 1666725954.8959005,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_12.state')},
                       'cmd_120': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733558,\n'
                                             '  "timestamp_ms" : '
                                             '1666733558546,\n'
                                             '  "time" : "Tue Oct 25 23:32:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3217563648,\n'
                                             '        "io_kbytes" : 3142152,\n'
                                             '        "bw_bytes" : 459586294,\n'
                                             '        "bw" : 448814,\n'
                                             '        "iops" : 112203.542351,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 785537,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 661,\n'
                                             '          "max" : 7804,\n'
                                             '          "mean" : 916.427430,\n'
                                             '          "stddev" : 81.567851,\n'
                                             '          "N" : 785537\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 7284,\n'
                                             '          "max" : 134732,\n'
                                             '          "mean" : '
                                             '16623.412960,\n'
                                             '          "stddev" : '
                                             '3660.577461,\n'
                                             '          "N" : 785538,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13632,\n'
                                             '            "5.000000" : 13888,\n'
                                             '            "10.000000" : '
                                             '14016,\n'
                                             '            "20.000000" : '
                                             '14144,\n'
                                             '            "30.000000" : '
                                             '14272,\n'
                                             '            "40.000000" : '
                                             '14400,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '17792,\n'
                                             '            "80.000000" : '
                                             '18048,\n'
                                             '            "90.000000" : '
                                             '18048,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '30848,\n'
                                             '            "99.500000" : '
                                             '34560,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 79360\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11752,\n'
                                             '          "max" : 135514,\n'
                                             '          "mean" : '
                                             '17539.840385,\n'
                                             '          "stddev" : '
                                             '3659.715140,\n'
                                             '          "N" : 785538\n'
                                             '        },\n'
                                             '        "bw_min" : 447328,\n'
                                             '        "bw_max" : 451014,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '448941.000000,\n'
                                             '        "bw_dev" : 1118.741257,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 111832,\n'
                                             '        "iops_max" : 112753,\n'
                                             '        "iops_mean" : '
                                             '112235.357143,\n'
                                             '        "iops_stddev" : '
                                             '279.439991,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 2,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 99.983196,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.054466,\n'
                                             '        "50" : 4.836819,\n'
                                             '        "100" : 0.108334,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=86080: Tue Oct 25 23:32:38 '
                                             '2022\n'
                                             '  read: IOPS=112k, BW=438MiB/s '
                                             '(460MB/s)(3069MiB/7001msec)\n'
                                             '    slat (nsec): min=661, '
                                             'max=7804, avg=916.43, '
                                             'stdev=81.57\n'
                                             '    clat (usec): min=7, max=134, '
                                             'avg=16.62, stdev= 3.66\n'
                                             '     lat (usec): min=11, '
                                             'max=135, avg=17.54, stdev= 3.66\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13632],  '
                                             '5.00th=[13888], 10.00th=[14016], '
                                             '20.00th=[14144],\n'
                                             '     | 30.00th=[14272], '
                                             '40.00th=[14400], '
                                             '50.00th=[17280], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[17792], '
                                             '80.00th=[18048], '
                                             '90.00th=[18048], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[30848], '
                                             '99.50th=[34560], '
                                             '99.90th=[51456], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[79360]\n'
                                             '   bw (  KiB/s): min=447328, '
                                             'max=451014, per=100.00%, '
                                             'avg=448941.00, stdev=1118.74, '
                                             'samples=14\n'
                                             '   iops        : min=111832, '
                                             'max=112753, avg=112235.36, '
                                             'stdev=279.44, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=95.05%, 50=4.84%, 100=0.11%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=2, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=785537,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=438MiB/s (460MB/s), '
                                             '438MiB/s-438MiB/s '
                                             '(460MB/s-460MB/s), io=3069MiB '
                                             '(3218MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.output'),
                                   'state': {'begin': 1666726469.1268544,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7891857624053955,
                                             'end': 1666726469.9160402,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_120.state')},
                       'cmd_121': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.output'),
                                   'state': {'begin': 1666726471.97713,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7857165336608887,
                                             'end': 1666726472.7628465,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_121.state')},
                       'cmd_122': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.output'),
                                   'state': {'begin': 1666726472.76527,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.503538608551025,
                                             'end': 1666726484.2688086,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_122.state')},
                       'cmd_123': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733573,\n'
                                             '  "timestamp_ms" : '
                                             '1666733573691,\n'
                                             '  "time" : "Tue Oct 25 23:32:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3401510912,\n'
                                             '        "io_kbytes" : 3321788,\n'
                                             '        "bw_bytes" : 485860721,\n'
                                             '        "bw" : 474473,\n'
                                             '        "iops" : 118618.197400,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 830446,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 671,\n'
                                             '          "max" : 11402,\n'
                                             '          "mean" : 967.528109,\n'
                                             '          "stddev" : '
                                             '130.573892,\n'
                                             '          "N" : 830446\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 7574,\n'
                                             '          "max" : 176641,\n'
                                             '          "mean" : '
                                             '15406.331199,\n'
                                             '          "stddev" : '
                                             '3395.306415,\n'
                                             '          "N" : 830447,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13632,\n'
                                             '            "5.000000" : 13760,\n'
                                             '            "10.000000" : '
                                             '13888,\n'
                                             '            "20.000000" : '
                                             '14656,\n'
                                             '            "30.000000" : '
                                             '14784,\n'
                                             '            "40.000000" : '
                                             '14784,\n'
                                             '            "50.000000" : '
                                             '14912,\n'
                                             '            "60.000000" : '
                                             '15040,\n'
                                             '            "70.000000" : '
                                             '15040,\n'
                                             '            "80.000000" : '
                                             '15168,\n'
                                             '            "90.000000" : '
                                             '15296,\n'
                                             '            "95.000000" : '
                                             '15808,\n'
                                             '            "99.000000" : '
                                             '31104,\n'
                                             '            "99.500000" : '
                                             '36096,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 82432\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10650,\n'
                                             '          "max" : 177563,\n'
                                             '          "mean" : '
                                             '16373.859542,\n'
                                             '          "stddev" : '
                                             '3397.897104,\n'
                                             '          "N" : 830447\n'
                                             '        },\n'
                                             '        "bw_min" : 473232,\n'
                                             '        "bw_max" : 475744,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '474607.785714,\n'
                                             '        "bw_dev" : 803.230801,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 118308,\n'
                                             '        "iops_max" : 118936,\n'
                                             '        "iops_mean" : '
                                             '118651.928571,\n'
                                             '        "iops_stddev" : '
                                             '200.807778,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 7,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 15.182565,\n'
                                             '        "2" : 84.817435,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.736147,\n'
                                             '        "50" : 4.161378,\n'
                                             '        "100" : 0.099224,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=86130: Tue Oct 25 23:32:53 '
                                             '2022\n'
                                             '  read: IOPS=119k, BW=463MiB/s '
                                             '(486MB/s)(3244MiB/7001msec)\n'
                                             '    slat (nsec): min=671, '
                                             'max=11402, avg=967.53, '
                                             'stdev=130.57\n'
                                             '    clat (usec): min=7, max=176, '
                                             'avg=15.41, stdev= 3.40\n'
                                             '     lat (usec): min=10, '
                                             'max=177, avg=16.37, stdev= 3.40\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13632],  '
                                             '5.00th=[13760], 10.00th=[13888], '
                                             '20.00th=[14656],\n'
                                             '     | 30.00th=[14784], '
                                             '40.00th=[14784], '
                                             '50.00th=[14912], '
                                             '60.00th=[15040],\n'
                                             '     | 70.00th=[15040], '
                                             '80.00th=[15168], '
                                             '90.00th=[15296], '
                                             '95.00th=[15808],\n'
                                             '     | 99.00th=[31104], '
                                             '99.50th=[36096], '
                                             '99.90th=[51456], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[82432]\n'
                                             '   bw (  KiB/s): min=473232, '
                                             'max=475744, per=100.00%, '
                                             'avg=474607.79, stdev=803.23, '
                                             'samples=14\n'
                                             '   iops        : min=118308, '
                                             'max=118936, avg=118651.93, '
                                             'stdev=200.81, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=95.74%, 50=4.16%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=7, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=15.2%, '
                                             '2=84.8%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=830446,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=463MiB/s (486MB/s), '
                                             '463MiB/s-463MiB/s '
                                             '(486MB/s-486MB/s), io=3244MiB '
                                             '(3402MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.output'),
                                   'state': {'begin': 1666726484.271526,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7860622406005859,
                                             'end': 1666726485.0575883,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_123.state')},
                       'cmd_124': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.output'),
                                   'state': {'begin': 1666726487.1294892,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7789244651794434,
                                             'end': 1666726487.9084136,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_124.state')},
                       'cmd_125': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.output'),
                                   'state': {'begin': 1666726487.9108245,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.484877109527588,
                                             'end': 1666726499.3957016,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_125.state')},
                       'cmd_126': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733588,\n'
                                             '  "timestamp_ms" : '
                                             '1666733588818,\n'
                                             '  "time" : "Tue Oct 25 23:33:08 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3410411520,\n'
                                             '        "io_kbytes" : 3330480,\n'
                                             '        "bw_bytes" : 487132055,\n'
                                             '        "bw" : 475714,\n'
                                             '        "iops" : 118928.581631,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 832619,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 661,\n'
                                             '          "max" : 50916,\n'
                                             '          "mean" : 946.325844,\n'
                                             '          "stddev" : '
                                             '169.673091,\n'
                                             '          "N" : 832619\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 881,\n'
                                             '          "max" : 1304386,\n'
                                             '          "mean" : '
                                             '15275.264800,\n'
                                             '          "stddev" : '
                                             '3935.692281,\n'
                                             '          "N" : 832620,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13504,\n'
                                             '            "5.000000" : 13632,\n'
                                             '            "10.000000" : '
                                             '13760,\n'
                                             '            "20.000000" : '
                                             '13888,\n'
                                             '            "30.000000" : '
                                             '14656,\n'
                                             '            "40.000000" : '
                                             '14784,\n'
                                             '            "50.000000" : '
                                             '14784,\n'
                                             '            "60.000000" : '
                                             '14912,\n'
                                             '            "70.000000" : '
                                             '15040,\n'
                                             '            "80.000000" : '
                                             '15040,\n'
                                             '            "90.000000" : '
                                             '15296,\n'
                                             '            "95.000000" : '
                                             '15808,\n'
                                             '            "99.000000" : '
                                             '31104,\n'
                                             '            "99.500000" : '
                                             '36608,\n'
                                             '            "99.900000" : '
                                             '48896,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 82432\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10650,\n'
                                             '          "max" : 1305568,\n'
                                             '          "mean" : '
                                             '16221.590603,\n'
                                             '          "stddev" : '
                                             '3939.658232,\n'
                                             '          "N" : 832620\n'
                                             '        },\n'
                                             '        "bw_min" : 474520,\n'
                                             '        "bw_max" : 477072,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '475836.000000,\n'
                                             '        "bw_dev" : 739.461759,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 118630,\n'
                                             '        "iops_max" : 119268,\n'
                                             '        "iops_mean" : '
                                             '118959.000000,\n'
                                             '        "iops_stddev" : '
                                             '184.865440,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 19,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 25.146916,\n'
                                             '        "2" : 74.853084,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.767812,\n'
                                             '        "50" : 4.132622,\n'
                                             '        "100" : 0.094521,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=86180: Tue Oct 25 23:33:08 '
                                             '2022\n'
                                             '  read: IOPS=119k, BW=465MiB/s '
                                             '(487MB/s)(3252MiB/7001msec)\n'
                                             '    slat (nsec): min=661, '
                                             'max=50916, avg=946.33, '
                                             'stdev=169.67\n'
                                             '    clat (nsec): min=881, '
                                             'max=1304.4k, avg=15275.26, '
                                             'stdev=3935.69\n'
                                             '     lat (usec): min=10, '
                                             'max=1305, avg=16.22, stdev= '
                                             '3.94\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13504],  '
                                             '5.00th=[13632], 10.00th=[13760], '
                                             '20.00th=[13888],\n'
                                             '     | 30.00th=[14656], '
                                             '40.00th=[14784], '
                                             '50.00th=[14784], '
                                             '60.00th=[14912],\n'
                                             '     | 70.00th=[15040], '
                                             '80.00th=[15040], '
                                             '90.00th=[15296], '
                                             '95.00th=[15808],\n'
                                             '     | 99.00th=[31104], '
                                             '99.50th=[36608], '
                                             '99.90th=[48896], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[82432]\n'
                                             '   bw (  KiB/s): min=474520, '
                                             'max=477072, per=100.00%, '
                                             'avg=475836.00, stdev=739.46, '
                                             'samples=14\n'
                                             '   iops        : min=118630, '
                                             'max=119268, avg=118959.00, '
                                             'stdev=184.87, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.01%, 20=95.77%, '
                                             '50=4.13%\n'
                                             '  lat (usec)   : 100=0.09%, '
                                             '250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=19, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=25.1%, '
                                             '2=74.9%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=832619,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=465MiB/s (487MB/s), '
                                             '465MiB/s-465MiB/s '
                                             '(487MB/s-487MB/s), io=3252MiB '
                                             '(3410MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.output'),
                                   'state': {'begin': 1666726499.3984802,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8138327598571777,
                                             'end': 1666726500.212313,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_126.state')},
                       'cmd_127': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.output'),
                                   'state': {'begin': 1666726501.2081702,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7935519218444824,
                                             'end': 1666726502.001722,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_127.state')},
                       'cmd_128': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.output'),
                                   'state': {'begin': 1666726502.0037568,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.046199321746826,
                                             'end': 1666726513.049956,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_128.state')},
                       'cmd_129': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733602,\n'
                                             '  "timestamp_ms" : '
                                             '1666733602544,\n'
                                             '  "time" : "Tue Oct 25 23:33:22 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2728185856,\n'
                                             '        "io_kbytes" : 2664244,\n'
                                             '        "bw_bytes" : 389685167,\n'
                                             '        "bw" : 380551,\n'
                                             '        "iops" : 95137.837452,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 666060,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2254,\n'
                                             '          "max" : 101570,\n'
                                             '          "mean" : 2532.068260,\n'
                                             '          "stddev" : '
                                             '388.305482,\n'
                                             '          "N" : 666060\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 150,\n'
                                             '          "max" : 157766,\n'
                                             '          "mean" : '
                                             '17905.455248,\n'
                                             '          "stddev" : '
                                             '3291.383779,\n'
                                             '          "N" : 666061,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17024,\n'
                                             '            "30.000000" : '
                                             '17024,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '17536,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '19840,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '40192,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63232,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11642,\n'
                                             '          "max" : 160081,\n'
                                             '          "mean" : '
                                             '20437.523182,\n'
                                             '          "stddev" : '
                                             '3295.392638,\n'
                                             '          "N" : 666061\n'
                                             '        },\n'
                                             '        "bw_min" : 379696,\n'
                                             '        "bw_max" : 382128,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '380707.500000,\n'
                                             '        "bw_dev" : 705.250065,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 94924,\n'
                                             '        "iops_max" : 95532,\n'
                                             '        "iops_mean" : '
                                             '95176.857143,\n'
                                             '        "iops_stddev" : '
                                             '176.305180,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 12.700000,\n'
                                             '      "sys_cpu" : 41.028571,\n'
                                             '      "ctx" : 332854,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.016215,\n'
                                             '        "20" : 95.018317,\n'
                                             '        "50" : 4.859622,\n'
                                             '        "100" : 0.102093,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 965358,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 12002,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 12002,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86219: Tue '
                                             'Oct 25 23:33:22 2022\n'
                                             '  read: IOPS=95.1k, BW=372MiB/s '
                                             '(390MB/s)(2602MiB/7001msec)\n'
                                             '    slat (nsec): min=2254, '
                                             'max=101570, avg=2532.07, '
                                             'stdev=388.31\n'
                                             '    clat (nsec): min=150, '
                                             'max=157766, avg=17905.46, '
                                             'stdev=3291.38\n'
                                             '     lat (usec): min=11, '
                                             'max=160, avg=20.44, stdev= 3.30\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[17024],\n'
                                             '     | 30.00th=[17024], '
                                             '40.00th=[17024], '
                                             '50.00th=[17280], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[17536], '
                                             '80.00th=[17536], '
                                             '90.00th=[17792], '
                                             '95.00th=[19840],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[40192], '
                                             '99.90th=[50432], '
                                             '99.95th=[63232],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=379696, '
                                             'max=382128, per=100.00%, '
                                             'avg=380707.50, stdev=705.25, '
                                             'samples=14\n'
                                             '   iops        : min=94924, '
                                             'max=95532, avg=95176.86, '
                                             'stdev=176.31, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.02%, 20=95.02%, 50=4.86%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=12.70%, '
                                             'sys=41.03%, ctx=332854, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=666060,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=372MiB/s (390MB/s), '
                                             '372MiB/s-372MiB/s '
                                             '(390MB/s-390MB/s), io=2602MiB '
                                             '(2728MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=965358/0, '
                                             'merge=0/0, ticks=12002/0, '
                                             'in_queue=12002, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.output'),
                                   'state': {'begin': 1666726513.051712,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7920506000518799,
                                             'end': 1666726513.8437626,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_129.state')},
                       'cmd_13': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.output'),
                                  'state': {'begin': 1666725955.8996396,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8654091358184814,
                                            'end': 1666725956.7650487,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_13.state')},
                       'cmd_130': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.output'),
                                   'state': {'begin': 1666726514.8286247,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7782762050628662,
                                             'end': 1666726515.606901,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_130.state')},
                       'cmd_131': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.output'),
                                   'state': {'begin': 1666726515.6089776,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.042916536331177,
                                             'end': 1666726526.651894,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_131.state')},
                       'cmd_132': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733616,\n'
                                             '  "timestamp_ms" : '
                                             '1666733616144,\n'
                                             '  "time" : "Tue Oct 25 23:33:36 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2763063296,\n'
                                             '        "io_kbytes" : 2698304,\n'
                                             '        "bw_bytes" : 394666947,\n'
                                             '        "bw" : 385416,\n'
                                             '        "iops" : 96354.092273,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 674575,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2134,\n'
                                             '          "max" : 30237,\n'
                                             '          "mean" : 2411.264627,\n'
                                             '          "stddev" : '
                                             '301.369070,\n'
                                             '          "N" : 674575\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3416,\n'
                                             '          "max" : 152055,\n'
                                             '          "mean" : '
                                             '17806.873363,\n'
                                             '          "stddev" : '
                                             '3321.266952,\n'
                                             '          "N" : 674576,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '17024,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17536,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '20096,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '40192,\n'
                                             '            "99.900000" : '
                                             '53504,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11702,\n'
                                             '          "max" : 154269,\n'
                                             '          "mean" : '
                                             '20218.137700,\n'
                                             '          "stddev" : '
                                             '3319.032476,\n'
                                             '          "N" : 674576\n'
                                             '        },\n'
                                             '        "bw_min" : 384640,\n'
                                             '        "bw_max" : 386432,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '385564.928571,\n'
                                             '        "bw_dev" : 441.118241,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 96160,\n'
                                             '        "iops_max" : 96608,\n'
                                             '        "iops_mean" : '
                                             '96391.142857,\n'
                                             '        "iops_stddev" : '
                                             '110.190644,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 11.157143,\n'
                                             '      "sys_cpu" : 41.785714,\n'
                                             '      "ctx" : 337155,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 94.990624,\n'
                                             '        "50" : 4.873884,\n'
                                             '        "100" : 0.124226,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 966533,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11927,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11927,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86257: Tue '
                                             'Oct 25 23:33:36 2022\n'
                                             '  read: IOPS=96.4k, BW=376MiB/s '
                                             '(395MB/s)(2635MiB/7001msec)\n'
                                             '    slat (nsec): min=2134, '
                                             'max=30237, avg=2411.26, '
                                             'stdev=301.37\n'
                                             '    clat (usec): min=3, max=152, '
                                             'avg=17.81, stdev= 3.32\n'
                                             '     lat (usec): min=11, '
                                             'max=154, avg=20.22, stdev= 3.32\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[17024], '
                                             '40.00th=[17024], '
                                             '50.00th=[17280], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17536], '
                                             '80.00th=[17536], '
                                             '90.00th=[17536], '
                                             '95.00th=[20096],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[40192], '
                                             '99.90th=[53504], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=384640, '
                                             'max=386432, per=100.00%, '
                                             'avg=385564.93, stdev=441.12, '
                                             'samples=14\n'
                                             '   iops        : min=96160, '
                                             'max=96608, avg=96391.14, '
                                             'stdev=110.19, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=94.99%, 50=4.87%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=11.16%, '
                                             'sys=41.79%, ctx=337155, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=674575,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=376MiB/s (395MB/s), '
                                             '376MiB/s-376MiB/s '
                                             '(395MB/s-395MB/s), io=2635MiB '
                                             '(2763MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=966533/0, '
                                             'merge=0/0, ticks=11927/0, '
                                             'in_queue=11927, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.output'),
                                   'state': {'begin': 1666726526.658701,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7847790718078613,
                                             'end': 1666726527.44348,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_132.state')},
                       'cmd_133': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.output'),
                                   'state': {'begin': 1666726528.439404,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7807440757751465,
                                             'end': 1666726529.220148,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_133.state')},
                       'cmd_134': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.output'),
                                   'state': {'begin': 1666726529.222193,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.053741455078125,
                                             'end': 1666726540.2759345,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_134.state')},
                       'cmd_135': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733629,\n'
                                             '  "timestamp_ms" : '
                                             '1666733629769,\n'
                                             '  "time" : "Tue Oct 25 23:33:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2769817600,\n'
                                             '        "io_kbytes" : 2704900,\n'
                                             '        "bw_bytes" : 395631709,\n'
                                             '        "bw" : 386359,\n'
                                             '        "iops" : 96589.630053,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 676224,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2174,\n'
                                             '          "max" : 56947,\n'
                                             '          "mean" : 2455.820520,\n'
                                             '          "stddev" : '
                                             '313.047630,\n'
                                             '          "N" : 676224\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3426,\n'
                                             '          "max" : 121939,\n'
                                             '          "mean" : '
                                             '17701.656814,\n'
                                             '          "stddev" : '
                                             '3232.617042,\n'
                                             '          "N" : 676225,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '16768,\n'
                                             '            "40.000000" : '
                                             '16768,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17280,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '17536,\n'
                                             '            "95.000000" : '
                                             '19840,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '36608,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11822,\n'
                                             '          "max" : 124183,\n'
                                             '          "mean" : '
                                             '20157.477096,\n'
                                             '          "stddev" : '
                                             '3230.430762,\n'
                                             '          "N" : 676225\n'
                                             '        },\n'
                                             '        "bw_min" : 385176,\n'
                                             '        "bw_max" : 387294,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '386515.642857,\n'
                                             '        "bw_dev" : 702.422359,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 96294,\n'
                                             '        "iops_max" : 96823,\n'
                                             '        "iops_mean" : '
                                             '96628.857143,\n'
                                             '        "iops_stddev" : '
                                             '175.563207,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 10.471429,\n'
                                             '      "sys_cpu" : 42.771429,\n'
                                             '      "ctx" : 337987,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 94.997516,\n'
                                             '        "50" : 4.886694,\n'
                                             '        "100" : 0.107509,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 967584,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11897,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11897,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86295: Tue '
                                             'Oct 25 23:33:49 2022\n'
                                             '  read: IOPS=96.6k, BW=377MiB/s '
                                             '(396MB/s)(2642MiB/7001msec)\n'
                                             '    slat (nsec): min=2174, '
                                             'max=56947, avg=2455.82, '
                                             'stdev=313.05\n'
                                             '    clat (usec): min=3, max=121, '
                                             'avg=17.70, stdev= 3.23\n'
                                             '     lat (usec): min=11, '
                                             'max=124, avg=20.16, stdev= 3.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16512],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[16768], '
                                             '40.00th=[16768], '
                                             '50.00th=[17280], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17280], '
                                             '80.00th=[17280], '
                                             '90.00th=[17536], '
                                             '95.00th=[19840],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[36608], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=385176, '
                                             'max=387294, per=100.00%, '
                                             'avg=386515.64, stdev=702.42, '
                                             'samples=14\n'
                                             '   iops        : min=96294, '
                                             'max=96823, avg=96628.86, '
                                             'stdev=175.56, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=95.00%, 50=4.89%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=10.47%, '
                                             'sys=42.77%, ctx=337987, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=676224,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=377MiB/s (396MB/s), '
                                             '377MiB/s-377MiB/s '
                                             '(396MB/s-396MB/s), io=2642MiB '
                                             '(2770MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=967584/0, '
                                             'merge=0/0, ticks=11897/0, '
                                             'in_queue=11897, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.output'),
                                   'state': {'begin': 1666726540.279404,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7870666980743408,
                                             'end': 1666726541.0664706,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_135.state')},
                       'cmd_136': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.output'),
                                   'state': {'begin': 1666726543.085222,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7838060855865479,
                                             'end': 1666726543.869028,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_136.state')},
                       'cmd_137': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.output'),
                                   'state': {'begin': 1666726543.8709989,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.33250117301941,
                                             'end': 1666726555.2035,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_137.state')},
                       'cmd_138': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733644,\n'
                                             '  "timestamp_ms" : '
                                             '1666733644623,\n'
                                             '  "time" : "Tue Oct 25 23:34:04 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3403218944,\n'
                                             '        "io_kbytes" : 3323456,\n'
                                             '        "bw_bytes" : 486104691,\n'
                                             '        "bw" : 474711,\n'
                                             '        "iops" : 118677.760320,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 830863,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 13305,\n'
                                             '          "mean" : 152.035344,\n'
                                             '          "stddev" : 44.246764,\n'
                                             '          "N" : 830863\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11011,\n'
                                             '          "max" : 125345,\n'
                                             '          "mean" : '
                                             '16229.591323,\n'
                                             '          "stddev" : '
                                             '3439.794272,\n'
                                             '          "N" : 830863,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15168,\n'
                                             '            "5.000000" : 15168,\n'
                                             '            "10.000000" : '
                                             '15296,\n'
                                             '            "20.000000" : '
                                             '15296,\n'
                                             '            "30.000000" : '
                                             '15424,\n'
                                             '            "40.000000" : '
                                             '15552,\n'
                                             '            "50.000000" : '
                                             '15552,\n'
                                             '            "60.000000" : '
                                             '15680,\n'
                                             '            "70.000000" : '
                                             '15680,\n'
                                             '            "80.000000" : '
                                             '15808,\n'
                                             '            "90.000000" : '
                                             '15936,\n'
                                             '            "95.000000" : '
                                             '16512,\n'
                                             '            "99.000000" : '
                                             '31872,\n'
                                             '            "99.500000" : '
                                             '37632,\n'
                                             '            "99.900000" : '
                                             '49920,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 84480\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11191,\n'
                                             '          "max" : 125525,\n'
                                             '          "mean" : '
                                             '16381.626666,\n'
                                             '          "stddev" : '
                                             '3441.514411,\n'
                                             '          "N" : 830863\n'
                                             '        },\n'
                                             '        "bw_min" : 473656,\n'
                                             '        "bw_max" : 476112,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '474806.214286,\n'
                                             '        "bw_dev" : 872.970718,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 118414,\n'
                                             '        "iops_max" : 119028,\n'
                                             '        "iops_mean" : '
                                             '118701.214286,\n'
                                             '        "iops_stddev" : '
                                             '218.461818,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 77.185714,\n'
                                             '      "sys_cpu" : 22.785714,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.969128,\n'
                                             '        "2" : 50.030872,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 95.630567,\n'
                                             '        "50" : 4.272425,\n'
                                             '        "100" : 0.096526,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=86344: Tue '
                                             'Oct 25 23:34:04 2022\n'
                                             '  read: IOPS=119k, BW=464MiB/s '
                                             '(486MB/s)(3246MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=13305, avg=152.04, '
                                             'stdev=44.25\n'
                                             '    clat (usec): min=11, '
                                             'max=125, avg=16.23, stdev= 3.44\n'
                                             '     lat (usec): min=11, '
                                             'max=125, avg=16.38, stdev= 3.44\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[15168],  '
                                             '5.00th=[15168], 10.00th=[15296], '
                                             '20.00th=[15296],\n'
                                             '     | 30.00th=[15424], '
                                             '40.00th=[15552], '
                                             '50.00th=[15552], '
                                             '60.00th=[15680],\n'
                                             '     | 70.00th=[15680], '
                                             '80.00th=[15808], '
                                             '90.00th=[15936], '
                                             '95.00th=[16512],\n'
                                             '     | 99.00th=[31872], '
                                             '99.50th=[37632], '
                                             '99.90th=[49920], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[84480]\n'
                                             '   bw (  KiB/s): min=473656, '
                                             'max=476112, per=100.00%, '
                                             'avg=474806.21, stdev=872.97, '
                                             'samples=14\n'
                                             '   iops        : min=118414, '
                                             'max=119028, avg=118701.21, '
                                             'stdev=218.46, samples=14\n'
                                             '  lat (usec)   : 20=95.63%, '
                                             '50=4.27%, 100=0.10%, 250=0.01%\n'
                                             '  cpu          : usr=77.19%, '
                                             'sys=22.79%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=830863,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=464MiB/s (486MB/s), '
                                             '464MiB/s-464MiB/s '
                                             '(486MB/s-486MB/s), io=3246MiB '
                                             '(3403MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.output'),
                                   'state': {'begin': 1666726555.2066221,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7880933284759521,
                                             'end': 1666726555.9947155,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_138.state')},
                       'cmd_139': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.output'),
                                   'state': {'begin': 1666726558.1017845,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7905898094177246,
                                             'end': 1666726558.8923743,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_139.state')},
                       'cmd_14': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.output'),
                                  'state': {'begin': 1666725956.7671995,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.056231498718262,
                                            'end': 1666725967.823431,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_14.state')},
                       'cmd_140': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.output'),
                                   'state': {'begin': 1666726558.8945546,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.452940225601196,
                                             'end': 1666726570.3474948,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_140.state')},
                       'cmd_141': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733659,\n'
                                             '  "timestamp_ms" : '
                                             '1666733659767,\n'
                                             '  "time" : "Tue Oct 25 23:34:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3699757056,\n'
                                             '        "io_kbytes" : 3613044,\n'
                                             '        "bw_bytes" : 528461227,\n'
                                             '        "bw" : 516075,\n'
                                             '        "iops" : 129018.711613,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 903260,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 13215,\n'
                                             '          "mean" : 159.408986,\n'
                                             '          "stddev" : 56.703624,\n'
                                             '          "N" : 903260\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 9217,\n'
                                             '          "max" : 1316439,\n'
                                             '          "mean" : '
                                             '14838.287266,\n'
                                             '          "stddev" : '
                                             '4093.044611,\n'
                                             '          "N" : 903260,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13888,\n'
                                             '            "5.000000" : 13888,\n'
                                             '            "10.000000" : '
                                             '13888,\n'
                                             '            "20.000000" : '
                                             '14016,\n'
                                             '            "30.000000" : '
                                             '14016,\n'
                                             '            "40.000000" : '
                                             '14144,\n'
                                             '            "50.000000" : '
                                             '14144,\n'
                                             '            "60.000000" : '
                                             '14272,\n'
                                             '            "70.000000" : '
                                             '14272,\n'
                                             '            "80.000000" : '
                                             '14400,\n'
                                             '            "90.000000" : '
                                             '14400,\n'
                                             '            "95.000000" : '
                                             '15168,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '39168,\n'
                                             '            "99.900000" : '
                                             '48384,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 84480\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9407,\n'
                                             '          "max" : 1316629,\n'
                                             '          "mean" : '
                                             '14997.696252,\n'
                                             '          "stddev" : '
                                             '4094.615509,\n'
                                             '          "N" : 903260\n'
                                             '        },\n'
                                             '        "bw_min" : 514352,\n'
                                             '        "bw_max" : 517672,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '516195.357143,\n'
                                             '        "bw_dev" : 856.680679,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 128588,\n'
                                             '        "iops_max" : 129418,\n'
                                             '        "iops_mean" : '
                                             '129048.785714,\n'
                                             '        "iops_stddev" : '
                                             '214.201264,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 68.600000,\n'
                                             '      "sys_cpu" : 31.357143,\n'
                                             '      "ctx" : 50,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.986936,\n'
                                             '        "2" : 50.013064,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.020924,\n'
                                             '        "20" : 96.007240,\n'
                                             '        "50" : 3.875518,\n'
                                             '        "100" : 0.095654,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=86393: Tue '
                                             'Oct 25 23:34:19 2022\n'
                                             '  read: IOPS=129k, BW=504MiB/s '
                                             '(528MB/s)(3528MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=13215, avg=159.41, '
                                             'stdev=56.70\n'
                                             '    clat (usec): min=9, '
                                             'max=1316, avg=14.84, stdev= '
                                             '4.09\n'
                                             '     lat (usec): min=9, '
                                             'max=1316, avg=15.00, stdev= '
                                             '4.09\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[13888],  '
                                             '5.00th=[13888], 10.00th=[13888], '
                                             '20.00th=[14016],\n'
                                             '     | 30.00th=[14016], '
                                             '40.00th=[14144], '
                                             '50.00th=[14144], '
                                             '60.00th=[14272],\n'
                                             '     | 70.00th=[14272], '
                                             '80.00th=[14400], '
                                             '90.00th=[14400], '
                                             '95.00th=[15168],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[39168], '
                                             '99.90th=[48384], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[84480]\n'
                                             '   bw (  KiB/s): min=514352, '
                                             'max=517672, per=100.00%, '
                                             'avg=516195.36, stdev=856.68, '
                                             'samples=14\n'
                                             '   iops        : min=128588, '
                                             'max=129418, avg=129048.79, '
                                             'stdev=214.20, samples=14\n'
                                             '  lat (usec)   : 10=0.02%, '
                                             '20=96.01%, 50=3.88%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=68.60%, '
                                             'sys=31.36%, ctx=50, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=903260,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=504MiB/s (528MB/s), '
                                             '504MiB/s-504MiB/s '
                                             '(528MB/s-528MB/s), io=3528MiB '
                                             '(3700MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.output'),
                                   'state': {'begin': 1666726570.3504097,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7888486385345459,
                                             'end': 1666726571.1392584,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_141.state')},
                       'cmd_142': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.output'),
                                   'state': {'begin': 1666726573.2099755,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.794795036315918,
                                             'end': 1666726574.0047705,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_142.state')},
                       'cmd_143': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.output'),
                                   'state': {'begin': 1666726574.0118794,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.472287893295288,
                                             'end': 1666726585.4841673,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_143.state')},
                       'cmd_144': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733674,\n'
                                             '  "timestamp_ms" : '
                                             '1666733674905,\n'
                                             '  "time" : "Tue Oct 25 23:34:34 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3672137728,\n'
                                             '        "io_kbytes" : 3586072,\n'
                                             '        "bw_bytes" : 524516173,\n'
                                             '        "bw" : 512222,\n'
                                             '        "iops" : 128055.563491,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 896517,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 100,\n'
                                             '          "max" : 16821,\n'
                                             '          "mean" : 151.655918,\n'
                                             '          "stddev" : 54.676795,\n'
                                             '          "N" : 896517\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 9408,\n'
                                             '          "max" : 102191,\n'
                                             '          "mean" : '
                                             '14971.500404,\n'
                                             '          "stddev" : '
                                             '3599.731107,\n'
                                             '          "N" : 896517,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 14016,\n'
                                             '            "10.000000" : '
                                             '14144,\n'
                                             '            "20.000000" : '
                                             '14144,\n'
                                             '            "30.000000" : '
                                             '14144,\n'
                                             '            "40.000000" : '
                                             '14272,\n'
                                             '            "50.000000" : '
                                             '14272,\n'
                                             '            "60.000000" : '
                                             '14400,\n'
                                             '            "70.000000" : '
                                             '14400,\n'
                                             '            "80.000000" : '
                                             '14400,\n'
                                             '            "90.000000" : '
                                             '14528,\n'
                                             '            "95.000000" : '
                                             '15296,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '39168,\n'
                                             '            "99.900000" : '
                                             '53504,\n'
                                             '            "99.950000" : '
                                             '67072,\n'
                                             '            "99.990000" : 82432\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9578,\n'
                                             '          "max" : 102372,\n'
                                             '          "mean" : '
                                             '15123.156323,\n'
                                             '          "stddev" : '
                                             '3601.418454,\n'
                                             '          "N" : 896517\n'
                                             '        },\n'
                                             '        "bw_min" : 511288,\n'
                                             '        "bw_max" : 513200,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '512352.071429,\n'
                                             '        "bw_dev" : 676.281287,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 127820,\n'
                                             '        "iops_max" : 128300,\n'
                                             '        "iops_mean" : '
                                             '128088.000000,\n'
                                             '        "iops_stddev" : '
                                             '169.357156,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 64.900000,\n'
                                             '      "sys_cpu" : 35.071429,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.996933,\n'
                                             '        "2" : 50.003067,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.756801,\n'
                                             '        "50" : 4.139018,\n'
                                             '        "100" : 0.101950,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=86443: Tue '
                                             'Oct 25 23:34:34 2022\n'
                                             '  read: IOPS=128k, BW=500MiB/s '
                                             '(525MB/s)(3502MiB/7001msec)\n'
                                             '    slat (nsec): min=100, '
                                             'max=16821, avg=151.66, '
                                             'stdev=54.68\n'
                                             '    clat (usec): min=9, max=102, '
                                             'avg=14.97, stdev= 3.60\n'
                                             '     lat (usec): min=9, max=102, '
                                             'avg=15.12, stdev= 3.60\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[14016],  '
                                             '5.00th=[14016], 10.00th=[14144], '
                                             '20.00th=[14144],\n'
                                             '     | 30.00th=[14144], '
                                             '40.00th=[14272], '
                                             '50.00th=[14272], '
                                             '60.00th=[14400],\n'
                                             '     | 70.00th=[14400], '
                                             '80.00th=[14400], '
                                             '90.00th=[14528], '
                                             '95.00th=[15296],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[39168], '
                                             '99.90th=[53504], '
                                             '99.95th=[67072],\n'
                                             '     | 99.99th=[82432]\n'
                                             '   bw (  KiB/s): min=511288, '
                                             'max=513200, per=100.00%, '
                                             'avg=512352.07, stdev=676.28, '
                                             'samples=14\n'
                                             '   iops        : min=127820, '
                                             'max=128300, avg=128088.00, '
                                             'stdev=169.36, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=95.76%, 50=4.14%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=64.90%, '
                                             'sys=35.07%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=896517,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=500MiB/s (525MB/s), '
                                             '500MiB/s-500MiB/s '
                                             '(525MB/s-525MB/s), io=3502MiB '
                                             '(3672MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.output'),
                                   'state': {'begin': 1666726585.4879289,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7787735462188721,
                                             'end': 1666726586.2667024,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_144.state')},
                       'cmd_145': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.output'),
                                   'state': {'begin': 1666726588.3478794,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863411903381348,
                                             'end': 1666726589.1342206,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_145.state')},
                       'cmd_146': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.output'),
                                   'state': {'begin': 1666726589.1363273,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.435516595840454,
                                             'end': 1666726600.5718439,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_146.state')},
                       'cmd_147': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733689,\n'
                                             '  "timestamp_ms" : '
                                             '1666733689991,\n'
                                             '  "time" : "Tue Oct 25 23:34:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3760844800,\n'
                                             '        "io_kbytes" : 3672700,\n'
                                             '        "bw_bytes" : 537186801,\n'
                                             '        "bw" : 524596,\n'
                                             '        "iops" : 131148.978717,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 918174,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1643,\n'
                                             '          "max" : 15720,\n'
                                             '          "mean" : 1814.378433,\n'
                                             '          "stddev" : '
                                             '122.167456,\n'
                                             '          "N" : 918174\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 261,\n'
                                             '          "max" : 95289,\n'
                                             '          "mean" : '
                                             '13203.321858,\n'
                                             '          "stddev" : '
                                             '4291.070630,\n'
                                             '          "N" : 918175,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10048,\n'
                                             '            "5.000000" : 10176,\n'
                                             '            "10.000000" : '
                                             '10176,\n'
                                             '            "20.000000" : '
                                             '10176,\n'
                                             '            "30.000000" : '
                                             '10304,\n'
                                             '            "40.000000" : '
                                             '10304,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '14656,\n'
                                             '            "70.000000" : '
                                             '14656,\n'
                                             '            "80.000000" : '
                                             '14784,\n'
                                             '            "90.000000" : '
                                             '14912,\n'
                                             '            "95.000000" : '
                                             '15552,\n'
                                             '            "99.000000" : '
                                             '31616,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '44800,\n'
                                             '            "99.950000" : '
                                             '62720,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9618,\n'
                                             '          "max" : 97092,\n'
                                             '          "mean" : '
                                             '15017.700203,\n'
                                             '          "stddev" : '
                                             '4292.850203,\n'
                                             '          "N" : 918175\n'
                                             '        },\n'
                                             '        "bw_min" : 523296,\n'
                                             '        "bw_max" : 526613,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '524883.928571,\n'
                                             '        "bw_dev" : 955.591357,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 130824,\n'
                                             '        "iops_max" : 131653,\n'
                                             '        "iops_mean" : '
                                             '131220.928571,\n'
                                             '        "iops_stddev" : '
                                             '238.846737,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 11.700000,\n'
                                             '      "sys_cpu" : 88.271429,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.690174,\n'
                                             '        "20" : 95.137632,\n'
                                             '        "50" : 4.081253,\n'
                                             '        "100" : 0.088110,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86494: Tue '
                                             'Oct 25 23:34:49 2022\n'
                                             '  read: IOPS=131k, BW=512MiB/s '
                                             '(537MB/s)(3587MiB/7001msec)\n'
                                             '    slat (nsec): min=1643, '
                                             'max=15720, avg=1814.38, '
                                             'stdev=122.17\n'
                                             '    clat (nsec): min=261, '
                                             'max=95289, avg=13203.32, '
                                             'stdev=4291.07\n'
                                             '     lat (nsec): min=9618, '
                                             'max=97092, avg=15017.70, '
                                             'stdev=4292.85\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10048],  '
                                             '5.00th=[10176], 10.00th=[10176], '
                                             '20.00th=[10176],\n'
                                             '     | 30.00th=[10304], '
                                             '40.00th=[10304], '
                                             '50.00th=[14528], '
                                             '60.00th=[14656],\n'
                                             '     | 70.00th=[14656], '
                                             '80.00th=[14784], '
                                             '90.00th=[14912], '
                                             '95.00th=[15552],\n'
                                             '     | 99.00th=[31616], '
                                             '99.50th=[37120], '
                                             '99.90th=[44800], '
                                             '99.95th=[62720],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=523296, '
                                             'max=526613, per=100.00%, '
                                             'avg=524883.93, stdev=955.59, '
                                             'samples=14\n'
                                             '   iops        : min=130824, '
                                             'max=131653, avg=131220.93, '
                                             'stdev=238.85, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.69%, 20=95.14%, '
                                             '50=4.08%\n'
                                             '  lat (usec)   : 100=0.09%\n'
                                             '  cpu          : usr=11.70%, '
                                             'sys=88.27%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=918174,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=512MiB/s (537MB/s), '
                                             '512MiB/s-512MiB/s '
                                             '(537MB/s-537MB/s), io=3587MiB '
                                             '(3761MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.output'),
                                   'state': {'begin': 1666726600.5745914,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7963974475860596,
                                             'end': 1666726601.3709888,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_147.state')},
                       'cmd_148': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.output'),
                                   'state': {'begin': 1666726603.3925414,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7927045822143555,
                                             'end': 1666726604.185246,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_148.state')},
                       'cmd_149': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.output'),
                                   'state': {'begin': 1666726604.1870391,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.46225118637085,
                                             'end': 1666726615.6492903,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_149.state')},
                       'cmd_15': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733057,\n'
                                            '  "timestamp_ms" : '
                                            '1666733057326,\n'
                                            '  "time" : "Tue Oct 25 23:24:17 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1745297408,\n'
                                            '        "io_kbytes" : 1704392,\n'
                                            '        "bw_bytes" : 249292587,\n'
                                            '        "bw" : 243449,\n'
                                            '        "iops" : 60862.448222,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 426098,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2294,\n'
                                            '          "max" : 44142,\n'
                                            '          "mean" : 2645.758187,\n'
                                            '          "stddev" : 205.940659,\n'
                                            '          "N" : 426098\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 351,\n'
                                            '          "max" : 105057,\n'
                                            '          "mean" : 13250.876676,\n'
                                            '          "stddev" : '
                                            '2201.080057,\n'
                                            '          "N" : 426098,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 12096,\n'
                                            '            "5.000000" : 12096,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12224,\n'
                                            '            "30.000000" : 12352,\n'
                                            '            "40.000000" : 13248,\n'
                                            '            "50.000000" : 13376,\n'
                                            '            "60.000000" : 13376,\n'
                                            '            "70.000000" : 13504,\n'
                                            '            "80.000000" : 13504,\n'
                                            '            "90.000000" : 13504,\n'
                                            '            "95.000000" : 13632,\n'
                                            '            "99.000000" : 22912,\n'
                                            '            "99.500000" : 29312,\n'
                                            '            "99.900000" : 38656,\n'
                                            '            "99.950000" : 52992,\n'
                                            '            "99.990000" : 69120\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 14187,\n'
                                            '          "max" : 114945,\n'
                                            '          "mean" : 15896.634863,\n'
                                            '          "stddev" : '
                                            '2223.085816,\n'
                                            '          "N" : 426098\n'
                                            '        },\n'
                                            '        "bw_min" : 235512,\n'
                                            '        "bw_max" : 257448,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '243501.285714,\n'
                                            '        "bw_dev" : 9024.250392,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 58878,\n'
                                            '        "iops_max" : 64362,\n'
                                            '        "iops_mean" : '
                                            '60875.285714,\n'
                                            '        "iops_stddev" : '
                                            '2256.088870,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.300000,\n'
                                            '      "sys_cpu" : 34.514286,\n'
                                            '      "ctx" : 426091,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.344982,\n'
                                            '        "50" : 1.589306,\n'
                                            '        "100" : 0.061488,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=84545: Tue '
                                            'Oct 25 23:24:17 2022\n'
                                            '  read: IOPS=60.9k, BW=238MiB/s '
                                            '(249MB/s)(1664MiB/7001msec)\n'
                                            '    slat (nsec): min=2294, '
                                            'max=44142, avg=2645.76, '
                                            'stdev=205.94\n'
                                            '    clat (nsec): min=351, '
                                            'max=105057, avg=13250.88, '
                                            'stdev=2201.08\n'
                                            '     lat (usec): min=14, max=114, '
                                            'avg=15.90, stdev= 2.22\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[12096],  '
                                            '5.00th=[12096], 10.00th=[12096], '
                                            '20.00th=[12224],\n'
                                            '     | 30.00th=[12352], '
                                            '40.00th=[13248], 50.00th=[13376], '
                                            '60.00th=[13376],\n'
                                            '     | 70.00th=[13504], '
                                            '80.00th=[13504], 90.00th=[13504], '
                                            '95.00th=[13632],\n'
                                            '     | 99.00th=[22912], '
                                            '99.50th=[29312], 99.90th=[38656], '
                                            '99.95th=[52992],\n'
                                            '     | 99.99th=[69120]\n'
                                            '   bw (  KiB/s): min=235512, '
                                            'max=257448, per=100.00%, '
                                            'avg=243501.29, stdev=9024.25, '
                                            'samples=14\n'
                                            '   iops        : min=58878, '
                                            'max=64362, avg=60875.29, '
                                            'stdev=2256.09, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.34%, 50=1.59%, 100=0.06%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=9.30%, '
                                            'sys=34.51%, ctx=426091, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=426098,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=238MiB/s (249MB/s), '
                                            '238MiB/s-238MiB/s '
                                            '(249MB/s-249MB/s), io=1664MiB '
                                            '(1745MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.output'),
                                  'state': {'begin': 1666725967.826345,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7927157878875732,
                                            'end': 1666725968.6190608,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_15.state')},
                       'cmd_150': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733705,\n'
                                             '  "timestamp_ms" : '
                                             '1666733705068,\n'
                                             '  "time" : "Tue Oct 25 23:35:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3756642304,\n'
                                             '        "io_kbytes" : 3668596,\n'
                                             '        "bw_bytes" : 536586531,\n'
                                             '        "bw" : 524010,\n'
                                             '        "iops" : 131002.428225,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 917148,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1643,\n'
                                             '          "max" : 19396,\n'
                                             '          "mean" : 1811.534450,\n'
                                             '          "stddev" : '
                                             '120.874810,\n'
                                             '          "N" : 917148\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 591,\n'
                                             '          "max" : 102302,\n'
                                             '          "mean" : '
                                             '13208.573923,\n'
                                             '          "stddev" : '
                                             '4374.320123,\n'
                                             '          "N" : 917149,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10048,\n'
                                             '            "5.000000" : 10176,\n'
                                             '            "10.000000" : '
                                             '10176,\n'
                                             '            "20.000000" : '
                                             '10176,\n'
                                             '            "30.000000" : '
                                             '10304,\n'
                                             '            "40.000000" : '
                                             '10304,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '14656,\n'
                                             '            "70.000000" : '
                                             '14656,\n'
                                             '            "80.000000" : '
                                             '14784,\n'
                                             '            "90.000000" : '
                                             '14912,\n'
                                             '            "95.000000" : '
                                             '15552,\n'
                                             '            "99.000000" : '
                                             '31872,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9618,\n'
                                             '          "max" : 104076,\n'
                                             '          "mean" : '
                                             '15020.108310,\n'
                                             '          "stddev" : '
                                             '4377.112921,\n'
                                             '          "N" : 917149\n'
                                             '        },\n'
                                             '        "bw_min" : 522248,\n'
                                             '        "bw_max" : 526256,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '524297.500000,\n'
                                             '        "bw_dev" : 1261.415673,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 130562,\n'
                                             '        "iops_max" : 131564,\n'
                                             '        "iops_mean" : '
                                             '131074.214286,\n'
                                             '        "iops_stddev" : '
                                             '315.322101,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 10.914286,\n'
                                             '      "sys_cpu" : 89.071429,\n'
                                             '      "ctx" : 17,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.643953,\n'
                                             '        "20" : 95.156507,\n'
                                             '        "50" : 4.090834,\n'
                                             '        "100" : 0.104999,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86544: Tue '
                                             'Oct 25 23:35:05 2022\n'
                                             '  read: IOPS=131k, BW=512MiB/s '
                                             '(537MB/s)(3583MiB/7001msec)\n'
                                             '    slat (nsec): min=1643, '
                                             'max=19396, avg=1811.53, '
                                             'stdev=120.87\n'
                                             '    clat (nsec): min=591, '
                                             'max=102302, avg=13208.57, '
                                             'stdev=4374.32\n'
                                             '     lat (usec): min=9, max=104, '
                                             'avg=15.02, stdev= 4.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10048],  '
                                             '5.00th=[10176], 10.00th=[10176], '
                                             '20.00th=[10176],\n'
                                             '     | 30.00th=[10304], '
                                             '40.00th=[10304], '
                                             '50.00th=[14528], '
                                             '60.00th=[14656],\n'
                                             '     | 70.00th=[14656], '
                                             '80.00th=[14784], '
                                             '90.00th=[14912], '
                                             '95.00th=[15552],\n'
                                             '     | 99.00th=[31872], '
                                             '99.50th=[37120], '
                                             '99.90th=[50432], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=522248, '
                                             'max=526256, per=100.00%, '
                                             'avg=524297.50, stdev=1261.42, '
                                             'samples=14\n'
                                             '   iops        : min=130562, '
                                             'max=131564, avg=131074.21, '
                                             'stdev=315.32, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.64%, 20=95.16%, '
                                             '50=4.09%\n'
                                             '  lat (usec)   : 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=10.91%, '
                                             'sys=89.07%, ctx=17, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=917148,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=512MiB/s (537MB/s), '
                                             '512MiB/s-512MiB/s '
                                             '(537MB/s-537MB/s), io=3583MiB '
                                             '(3757MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.output'),
                                   'state': {'begin': 1666726615.6520617,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7919247150421143,
                                             'end': 1666726616.4439864,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_150.state')},
                       'cmd_151': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.output'),
                                   'state': {'begin': 1666726618.4636512,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7874436378479004,
                                             'end': 1666726619.2510948,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_151.state')},
                       'cmd_152': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.output'),
                                   'state': {'begin': 1666726619.2533228,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.456940650939941,
                                             'end': 1666726630.7102635,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_152.state')},
                       'cmd_153': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733720,\n'
                                             '  "timestamp_ms" : '
                                             '1666733720129,\n'
                                             '  "time" : "Tue Oct 25 23:35:20 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3880730624,\n'
                                             '        "io_kbytes" : 3789776,\n'
                                             '        "bw_bytes" : 554310901,\n'
                                             '        "bw" : 541319,\n'
                                             '        "iops" : 135329.667190,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 947443,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1492,\n'
                                             '          "max" : 15409,\n'
                                             '          "mean" : 1579.608812,\n'
                                             '          "stddev" : '
                                             '127.933694,\n'
                                             '          "N" : 947443\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 270,\n'
                                             '          "max" : 118652,\n'
                                             '          "mean" : '
                                             '12956.637074,\n'
                                             '          "stddev" : '
                                             '3894.360504,\n'
                                             '          "N" : 947444,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10048,\n'
                                             '            "5.000000" : 10176,\n'
                                             '            "10.000000" : '
                                             '10176,\n'
                                             '            "20.000000" : '
                                             '11968,\n'
                                             '            "30.000000" : '
                                             '12096,\n'
                                             '            "40.000000" : '
                                             '12224,\n'
                                             '            "50.000000" : '
                                             '12224,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '12352,\n'
                                             '            "80.000000" : '
                                             '14144,\n'
                                             '            "90.000000" : '
                                             '14400,\n'
                                             '            "95.000000" : '
                                             '14656,\n'
                                             '            "99.000000" : '
                                             '30592,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '52480,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 75264\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9178,\n'
                                             '          "max" : 120225,\n'
                                             '          "mean" : '
                                             '14536.245858,\n'
                                             '          "stddev" : '
                                             '3892.786982,\n'
                                             '          "N" : 947444\n'
                                             '        },\n'
                                             '        "bw_min" : 540352,\n'
                                             '        "bw_max" : 542264,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '541450.857143,\n'
                                             '        "bw_dev" : 675.183719,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 135088,\n'
                                             '        "iops_max" : 135566,\n'
                                             '        "iops_mean" : '
                                             '135362.571429,\n'
                                             '        "iops_stddev" : '
                                             '168.735439,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 9.314286,\n'
                                             '      "sys_cpu" : 90.671429,\n'
                                             '      "ctx" : 7,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.089082,\n'
                                             '        "20" : 96.324423,\n'
                                             '        "50" : 3.473665,\n'
                                             '        "100" : 0.109769,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86593: Tue '
                                             'Oct 25 23:35:20 2022\n'
                                             '  read: IOPS=135k, BW=529MiB/s '
                                             '(554MB/s)(3701MiB/7001msec)\n'
                                             '    slat (nsec): min=1492, '
                                             'max=15409, avg=1579.61, '
                                             'stdev=127.93\n'
                                             '    clat (nsec): min=270, '
                                             'max=118652, avg=12956.64, '
                                             'stdev=3894.36\n'
                                             '     lat (usec): min=9, max=120, '
                                             'avg=14.54, stdev= 3.89\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10048],  '
                                             '5.00th=[10176], 10.00th=[10176], '
                                             '20.00th=[11968],\n'
                                             '     | 30.00th=[12096], '
                                             '40.00th=[12224], '
                                             '50.00th=[12224], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[12352], '
                                             '80.00th=[14144], '
                                             '90.00th=[14400], '
                                             '95.00th=[14656],\n'
                                             '     | 99.00th=[30592], '
                                             '99.50th=[37120], '
                                             '99.90th=[52480], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[75264]\n'
                                             '   bw (  KiB/s): min=540352, '
                                             'max=542264, per=100.00%, '
                                             'avg=541450.86, stdev=675.18, '
                                             'samples=14\n'
                                             '   iops        : min=135088, '
                                             'max=135566, avg=135362.57, '
                                             'stdev=168.74, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.09%, 20=96.32%, 50=3.47%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=9.31%, '
                                             'sys=90.67%, ctx=7, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=947443,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=529MiB/s (554MB/s), '
                                             '529MiB/s-529MiB/s '
                                             '(554MB/s-554MB/s), io=3701MiB '
                                             '(3881MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.output'),
                                   'state': {'begin': 1666726630.7130911,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7981386184692383,
                                             'end': 1666726631.5112298,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_153.state')},
                       'cmd_154': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.output'),
                                   'state': {'begin': 1666726632.5046132,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7883801460266113,
                                             'end': 1666726633.2929933,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_154.state')},
                       'cmd_155': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.output'),
                                   'state': {'begin': 1666726633.294968,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.063021183013916,
                                             'end': 1666726644.357989,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_155.state')},
                       'cmd_156': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733733,\n'
                                             '  "timestamp_ms" : '
                                             '1666733733849,\n'
                                             '  "time" : "Tue Oct 25 23:35:33 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2654441472,\n'
                                             '        "io_kbytes" : 2592228,\n'
                                             '        "bw_bytes" : 379151760,\n'
                                             '        "bw" : 370265,\n'
                                             '        "iops" : 92566.061991,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 648055,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2304,\n'
                                             '          "max" : 769522,\n'
                                             '          "mean" : 2713.917901,\n'
                                             '          "stddev" : '
                                             '997.462060,\n'
                                             '          "N" : 648056\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 341,\n'
                                             '          "max" : 775153,\n'
                                             '          "mean" : '
                                             '18398.808790,\n'
                                             '          "stddev" : '
                                             '3486.347443,\n'
                                             '          "N" : 648056,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '17024,\n'
                                             '            "20.000000" : '
                                             '17280,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '17792,\n'
                                             '            "50.000000" : '
                                             '17792,\n'
                                             '            "60.000000" : '
                                             '18048,\n'
                                             '            "70.000000" : '
                                             '18048,\n'
                                             '            "80.000000" : '
                                             '18304,\n'
                                             '            "90.000000" : '
                                             '18304,\n'
                                             '            "95.000000" : '
                                             '20864,\n'
                                             '            "99.000000" : '
                                             '34560,\n'
                                             '            "99.500000" : '
                                             '43776,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63232,\n'
                                             '            "99.990000" : 75264\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12253,\n'
                                             '          "max" : 792535,\n'
                                             '          "mean" : '
                                             '21112.726690,\n'
                                             '          "stddev" : '
                                             '3619.369648,\n'
                                             '          "N" : 648056\n'
                                             '        },\n'
                                             '        "bw_min" : 365080,\n'
                                             '        "bw_max" : 384096,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '370353.714286,\n'
                                             '        "bw_dev" : 8054.404657,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 91270,\n'
                                             '        "iops_max" : 96024,\n'
                                             '        "iops_mean" : '
                                             '92588.428571,\n'
                                             '        "iops_stddev" : '
                                             '2013.601164,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 11.171429,\n'
                                             '      "sys_cpu" : 40.400000,\n'
                                             '      "ctx" : 323954,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 94.840870,\n'
                                             '        "50" : 5.042628,\n'
                                             '        "100" : 0.106010,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86632: Tue '
                                             'Oct 25 23:35:33 2022\n'
                                             '  read: IOPS=92.6k, BW=362MiB/s '
                                             '(379MB/s)(2531MiB/7001msec)\n'
                                             '    slat (nsec): min=2304, '
                                             'max=769522, avg=2713.92, '
                                             'stdev=997.46\n'
                                             '    clat (nsec): min=341, '
                                             'max=775153, avg=18398.81, '
                                             'stdev=3486.35\n'
                                             '     lat (usec): min=12, '
                                             'max=792, avg=21.11, stdev= 3.62\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[16768], 10.00th=[17024], '
                                             '20.00th=[17280],\n'
                                             '     | 30.00th=[17536], '
                                             '40.00th=[17792], '
                                             '50.00th=[17792], '
                                             '60.00th=[18048],\n'
                                             '     | 70.00th=[18048], '
                                             '80.00th=[18304], '
                                             '90.00th=[18304], '
                                             '95.00th=[20864],\n'
                                             '     | 99.00th=[34560], '
                                             '99.50th=[43776], '
                                             '99.90th=[50432], '
                                             '99.95th=[63232],\n'
                                             '     | 99.99th=[75264]\n'
                                             '   bw (  KiB/s): min=365080, '
                                             'max=384096, per=100.00%, '
                                             'avg=370353.71, stdev=8054.40, '
                                             'samples=14\n'
                                             '   iops        : min=91270, '
                                             'max=96024, avg=92588.43, '
                                             'stdev=2013.60, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=94.84%, 50=5.04%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%, '
                                             '1000=0.01%\n'
                                             '  cpu          : usr=11.17%, '
                                             'sys=40.40%, ctx=323954, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=648055,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=362MiB/s (379MB/s), '
                                             '362MiB/s-362MiB/s '
                                             '(379MB/s-379MB/s), io=2531MiB '
                                             '(2654MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.output'),
                                   'state': {'begin': 1666726644.3610137,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7959699630737305,
                                             'end': 1666726645.1569836,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_156.state')},
                       'cmd_157': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.output'),
                                   'state': {'begin': 1666726646.1621208,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7792618274688721,
                                             'end': 1666726646.9413826,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_157.state')},
                       'cmd_158': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.output'),
                                   'state': {'begin': 1666726646.9433987,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.05719804763794,
                                             'end': 1666726658.0005968,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_158.state')},
                       'cmd_159': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733747,\n'
                                             '  "timestamp_ms" : '
                                             '1666733747490,\n'
                                             '  "time" : "Tue Oct 25 23:35:47 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2740174848,\n'
                                             '        "io_kbytes" : 2675952,\n'
                                             '        "bw_bytes" : 391397635,\n'
                                             '        "bw" : 382224,\n'
                                             '        "iops" : 95555.920583,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 668987,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2274,\n'
                                             '          "max" : 30738,\n'
                                             '          "mean" : 2583.744357,\n'
                                             '          "stddev" : '
                                             '358.833559,\n'
                                             '          "N" : 668987\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 430,\n'
                                             '          "max" : 156663,\n'
                                             '          "mean" : '
                                             '17860.045107,\n'
                                             '          "stddev" : '
                                             '3321.104405,\n'
                                             '          "N" : 668988,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17024,\n'
                                             '            "30.000000" : '
                                             '17024,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17280,\n'
                                             '            "70.000000" : '
                                             '17536,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '40704,\n'
                                             '            "99.900000" : '
                                             '51456,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12112,\n'
                                             '          "max" : 159028,\n'
                                             '          "mean" : '
                                             '20443.789060,\n'
                                             '          "stddev" : '
                                             '3321.208162,\n'
                                             '          "N" : 668988\n'
                                             '        },\n'
                                             '        "bw_min" : 380880,\n'
                                             '        "bw_max" : 384376,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '382313.500000,\n'
                                             '        "bw_dev" : 1209.400917,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 95220,\n'
                                             '        "iops_max" : 96094,\n'
                                             '        "iops_mean" : '
                                             '95578.357143,\n'
                                             '        "iops_stddev" : '
                                             '302.342548,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.242857,\n'
                                             '      "sys_cpu" : 41.428571,\n'
                                             '      "ctx" : 334337,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.022474,\n'
                                             '        "50" : 4.850617,\n'
                                             '        "100" : 0.116744,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86671: Tue '
                                             'Oct 25 23:35:47 2022\n'
                                             '  read: IOPS=95.6k, BW=373MiB/s '
                                             '(391MB/s)(2613MiB/7001msec)\n'
                                             '    slat (nsec): min=2274, '
                                             'max=30738, avg=2583.74, '
                                             'stdev=358.83\n'
                                             '    clat (nsec): min=430, '
                                             'max=156663, avg=17860.05, '
                                             'stdev=3321.10\n'
                                             '     lat (usec): min=12, '
                                             'max=159, avg=20.44, stdev= 3.32\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[17024],\n'
                                             '     | 30.00th=[17024], '
                                             '40.00th=[17024], '
                                             '50.00th=[17280], '
                                             '60.00th=[17280],\n'
                                             '     | 70.00th=[17536], '
                                             '80.00th=[17536], '
                                             '90.00th=[17792], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[40704], '
                                             '99.90th=[51456], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=380880, '
                                             'max=384376, per=100.00%, '
                                             'avg=382313.50, stdev=1209.40, '
                                             'samples=14\n'
                                             '   iops        : min=95220, '
                                             'max=96094, avg=95578.36, '
                                             'stdev=302.34, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=95.02%, 50=4.85%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.24%, '
                                             'sys=41.43%, ctx=334337, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=668987,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=373MiB/s (391MB/s), '
                                             '373MiB/s-373MiB/s '
                                             '(391MB/s-391MB/s), io=2613MiB '
                                             '(2740MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.output'),
                                   'state': {'begin': 1666726658.007309,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7918438911437988,
                                             'end': 1666726658.7991529,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_159.state')},
                       'cmd_16': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.output'),
                                  'state': {'begin': 1666725969.607921,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.782355785369873,
                                            'end': 1666725970.3902767,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_16.state')},
                       'cmd_160': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.output'),
                                   'state': {'begin': 1666726659.7862983,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7929518222808838,
                                             'end': 1666726660.57925,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_160.state')},
                       'cmd_161': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.output'),
                                   'state': {'begin': 1666726660.5814507,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_io_uring-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.05913257598877,
                                             'end': 1666726671.6405833,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_161.state')},
                       'cmd_162': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733761,\n'
                                             '  "timestamp_ms" : '
                                             '1666733761130,\n'
                                             '  "time" : "Tue Oct 25 23:36:01 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2729132032,\n'
                                             '        "io_kbytes" : 2665168,\n'
                                             '        "bw_bytes" : 389820315,\n'
                                             '        "bw" : 380683,\n'
                                             '        "iops" : 95170.689901,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 666290,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2294,\n'
                                             '          "max" : 39635,\n'
                                             '          "mean" : 2615.013979,\n'
                                             '          "stddev" : '
                                             '376.495980,\n'
                                             '          "N" : 666291\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 370,\n'
                                             '          "max" : 191489,\n'
                                             '          "mean" : '
                                             '17882.856067,\n'
                                             '          "stddev" : '
                                             '3375.558001,\n'
                                             '          "N" : 666291,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '17024,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '17280,\n'
                                             '            "60.000000" : '
                                             '17536,\n'
                                             '            "70.000000" : '
                                             '17536,\n'
                                             '            "80.000000" : '
                                             '17536,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '19584,\n'
                                             '            "99.000000" : '
                                             '34048,\n'
                                             '            "99.500000" : '
                                             '43264,\n'
                                             '            "99.900000" : '
                                             '53504,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11221,\n'
                                             '          "max" : 193854,\n'
                                             '          "mean" : '
                                             '20497.870046,\n'
                                             '          "stddev" : '
                                             '3378.217884,\n'
                                             '          "N" : 666291\n'
                                             '        },\n'
                                             '        "bw_min" : 378240,\n'
                                             '        "bw_max" : 382600,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '380772.071429,\n'
                                             '        "bw_dev" : 1090.263798,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 94560,\n'
                                             '        "iops_max" : 95650,\n'
                                             '        "iops_mean" : '
                                             '95193.000000,\n'
                                             '        "iops_stddev" : '
                                             '272.562484,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.257143,\n'
                                             '      "sys_cpu" : 39.942857,\n'
                                             '      "ctx" : 332996,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 95.048703,\n'
                                             '        "50" : 4.824326,\n'
                                             '        "100" : 0.118717,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=86711: Tue '
                                             'Oct 25 23:36:01 2022\n'
                                             '  read: IOPS=95.2k, BW=372MiB/s '
                                             '(390MB/s)(2603MiB/7001msec)\n'
                                             '    slat (nsec): min=2294, '
                                             'max=39635, avg=2615.01, '
                                             'stdev=376.50\n'
                                             '    clat (nsec): min=370, '
                                             'max=191489, avg=17882.86, '
                                             'stdev=3375.56\n'
                                             '     lat (usec): min=11, '
                                             'max=193, avg=20.50, stdev= 3.38\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[16768],  '
                                             '5.00th=[16768], 10.00th=[16768], '
                                             '20.00th=[16768],\n'
                                             '     | 30.00th=[17024], '
                                             '40.00th=[17024], '
                                             '50.00th=[17280], '
                                             '60.00th=[17536],\n'
                                             '     | 70.00th=[17536], '
                                             '80.00th=[17536], '
                                             '90.00th=[17792], '
                                             '95.00th=[19584],\n'
                                             '     | 99.00th=[34048], '
                                             '99.50th=[43264], '
                                             '99.90th=[53504], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=378240, '
                                             'max=382600, per=100.00%, '
                                             'avg=380772.07, stdev=1090.26, '
                                             'samples=14\n'
                                             '   iops        : min=94560, '
                                             'max=95650, avg=95193.00, '
                                             'stdev=272.56, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=95.05%, 50=4.82%, '
                                             '100=0.12%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=15.26%, '
                                             'sys=39.94%, ctx=332996, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=666290,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=372MiB/s (390MB/s), '
                                             '372MiB/s-372MiB/s '
                                             '(390MB/s-390MB/s), io=2603MiB '
                                             '(2729MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.output'),
                                   'state': {'begin': 1666726671.643497,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7917177677154541,
                                             'end': 1666726672.4352148,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_162.state')},
                       'cmd_163': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.output'),
                                   'state': {'begin': 1666726673.4310272,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7887732982635498,
                                             'end': 1666726674.2198005,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_163.state')},
                       'cmd_164': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.output'),
                                   'state': {'begin': 1666726674.2219372,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.08357286453247,
                                             'end': 1666726685.30551,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_164.state')},
                       'cmd_165': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733774,\n'
                                             '  "timestamp_ms" : '
                                             '1666733774795,\n'
                                             '  "time" : "Tue Oct 25 23:36:14 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2756984832,\n'
                                             '        "io_kbytes" : 2692368,\n'
                                             '        "bw_bytes" : 393798719,\n'
                                             '        "bw" : 384569,\n'
                                             '        "iops" : 96142.122554,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 673091,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2394,\n'
                                             '          "max" : 29245,\n'
                                             '          "mean" : 2705.547284,\n'
                                             '          "stddev" : '
                                             '335.582093,\n'
                                             '          "N" : 673091\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 802,\n'
                                             '          "max" : 199354,\n'
                                             '          "mean" : '
                                             '17555.780315,\n'
                                             '          "stddev" : '
                                             '5106.894427,\n'
                                             '          "N" : 673092,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 12864,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '12864,\n'
                                             '            "30.000000" : '
                                             '12992,\n'
                                             '            "40.000000" : '
                                             '13120,\n'
                                             '            "50.000000" : '
                                             '20608,\n'
                                             '            "60.000000" : '
                                             '20864,\n'
                                             '            "70.000000" : '
                                             '20864,\n'
                                             '            "80.000000" : '
                                             '20864,\n'
                                             '            "90.000000" : '
                                             '21120,\n'
                                             '            "95.000000" : '
                                             '21632,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '38656,\n'
                                             '            "99.900000" : '
                                             '48896,\n'
                                             '            "99.950000" : '
                                             '61696,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11712,\n'
                                             '          "max" : 202269,\n'
                                             '          "mean" : '
                                             '20261.327762,\n'
                                             '          "stddev" : '
                                             '5287.935425,\n'
                                             '          "N" : 673092\n'
                                             '        },\n'
                                             '        "bw_min" : 383832,\n'
                                             '        "bw_max" : 385720,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '384720.071429,\n'
                                             '        "bw_dev" : 610.691722,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 95958,\n'
                                             '        "iops_max" : 96430,\n'
                                             '        "iops_mean" : '
                                             '96179.928571,\n'
                                             '        "iops_stddev" : '
                                             '152.704070,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.714286,\n'
                                             '      "sys_cpu" : 42.514286,\n'
                                             '      "ctx" : 336367,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 47.780909,\n'
                                             '        "50" : 52.115241,\n'
                                             '        "100" : 0.093895,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 963565,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11496,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11496,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86751: Tue '
                                             'Oct 25 23:36:14 2022\n'
                                             '  read: IOPS=96.1k, BW=376MiB/s '
                                             '(394MB/s)(2629MiB/7001msec)\n'
                                             '    slat (nsec): min=2394, '
                                             'max=29245, avg=2705.55, '
                                             'stdev=335.58\n'
                                             '    clat (nsec): min=802, '
                                             'max=199354, avg=17555.78, '
                                             'stdev=5106.89\n'
                                             '     lat (usec): min=11, '
                                             'max=202, avg=20.26, stdev= 5.29\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[12864], 10.00th=[12864], '
                                             '20.00th=[12864],\n'
                                             '     | 30.00th=[12992], '
                                             '40.00th=[13120], '
                                             '50.00th=[20608], '
                                             '60.00th=[20864],\n'
                                             '     | 70.00th=[20864], '
                                             '80.00th=[20864], '
                                             '90.00th=[21120], '
                                             '95.00th=[21632],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[38656], '
                                             '99.90th=[48896], '
                                             '99.95th=[61696],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=383832, '
                                             'max=385720, per=100.00%, '
                                             'avg=384720.07, stdev=610.69, '
                                             'samples=14\n'
                                             '   iops        : min=95958, '
                                             'max=96430, avg=96179.93, '
                                             'stdev=152.70, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=47.78%, 50=52.12%, '
                                             '100=0.09%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=14.71%, '
                                             'sys=42.51%, ctx=336367, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=673091,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=376MiB/s (394MB/s), '
                                             '376MiB/s-376MiB/s '
                                             '(394MB/s-394MB/s), io=2629MiB '
                                             '(2757MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=963565/0, '
                                             'merge=0/0, ticks=11496/0, '
                                             'in_queue=11496, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.output'),
                                   'state': {'begin': 1666726685.308098,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7916755676269531,
                                             'end': 1666726686.0997736,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_165.state')},
                       'cmd_166': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.output'),
                                   'state': {'begin': 1666726687.1028388,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7810490131378174,
                                             'end': 1666726687.8838878,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_166.state')},
                       'cmd_167': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.output'),
                                   'state': {'begin': 1666726687.8861325,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.079588413238525,
                                             'end': 1666726698.965721,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_167.state')},
                       'cmd_168': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733788,\n'
                                             '  "timestamp_ms" : '
                                             '1666733788456,\n'
                                             '  "time" : "Tue Oct 25 23:36:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2764947456,\n'
                                             '        "io_kbytes" : 2700144,\n'
                                             '        "bw_bytes" : 394936074,\n'
                                             '        "bw" : 385679,\n'
                                             '        "iops" : 96419.654335,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 675034,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2384,\n'
                                             '          "max" : 46998,\n'
                                             '          "mean" : 2698.317754,\n'
                                             '          "stddev" : '
                                             '340.659642,\n'
                                             '          "N" : 675035\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 882,\n'
                                             '          "max" : 136166,\n'
                                             '          "mean" : '
                                             '17524.151786,\n'
                                             '          "stddev" : '
                                             '5078.250149,\n'
                                             '          "N" : 675035,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 12864,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '12992,\n'
                                             '            "30.000000" : '
                                             '12992,\n'
                                             '            "40.000000" : '
                                             '13120,\n'
                                             '            "50.000000" : '
                                             '20608,\n'
                                             '            "60.000000" : '
                                             '20608,\n'
                                             '            "70.000000" : '
                                             '20864,\n'
                                             '            "80.000000" : '
                                             '20864,\n'
                                             '            "90.000000" : '
                                             '20864,\n'
                                             '            "95.000000" : '
                                             '21632,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '39168,\n'
                                             '            "99.900000" : '
                                             '52480,\n'
                                             '            "99.950000" : '
                                             '66048,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12653,\n'
                                             '          "max" : 139051,\n'
                                             '          "mean" : '
                                             '20222.470281,\n'
                                             '          "stddev" : '
                                             '5270.579236,\n'
                                             '          "N" : 675035\n'
                                             '        },\n'
                                             '        "bw_min" : 385120,\n'
                                             '        "bw_max" : 386536,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '385827.285714,\n'
                                             '        "bw_dev" : 381.399332,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 96280,\n'
                                             '        "iops_max" : 96634,\n'
                                             '        "iops_mean" : '
                                             '96456.714286,\n'
                                             '        "iops_stddev" : '
                                             '95.320051,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.000000,\n'
                                             '      "sys_cpu" : 43.514286,\n'
                                             '      "ctx" : 337349,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 47.800703,\n'
                                             '        "50" : 52.086265,\n'
                                             '        "100" : 0.105328,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 965663,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11565,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11565,\n'
                                             '      "util" : 99.049128\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86789: Tue '
                                             'Oct 25 23:36:28 2022\n'
                                             '  read: IOPS=96.4k, BW=377MiB/s '
                                             '(395MB/s)(2637MiB/7001msec)\n'
                                             '    slat (nsec): min=2384, '
                                             'max=46998, avg=2698.32, '
                                             'stdev=340.66\n'
                                             '    clat (nsec): min=882, '
                                             'max=136166, avg=17524.15, '
                                             'stdev=5078.25\n'
                                             '     lat (usec): min=12, '
                                             'max=139, avg=20.22, stdev= 5.27\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[12864], 10.00th=[12864], '
                                             '20.00th=[12992],\n'
                                             '     | 30.00th=[12992], '
                                             '40.00th=[13120], '
                                             '50.00th=[20608], '
                                             '60.00th=[20608],\n'
                                             '     | 70.00th=[20864], '
                                             '80.00th=[20864], '
                                             '90.00th=[20864], '
                                             '95.00th=[21632],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[39168], '
                                             '99.90th=[52480], '
                                             '99.95th=[66048],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=385120, '
                                             'max=386536, per=100.00%, '
                                             'avg=385827.29, stdev=381.40, '
                                             'samples=14\n'
                                             '   iops        : min=96280, '
                                             'max=96634, avg=96456.71, '
                                             'stdev=95.32, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=47.80%, 50=52.09%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=13.00%, '
                                             'sys=43.51%, ctx=337349, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=675034,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=377MiB/s (395MB/s), '
                                             '377MiB/s-377MiB/s '
                                             '(395MB/s-395MB/s), io=2637MiB '
                                             '(2765MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=965663/0, '
                                             'merge=0/0, ticks=11565/0, '
                                             'in_queue=11565, util=99.05%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.output'),
                                   'state': {'begin': 1666726698.9724402,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7907559871673584,
                                             'end': 1666726699.7631962,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_168.state')},
                       'cmd_169': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.output'),
                                   'state': {'begin': 1666726700.7536056,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7884535789489746,
                                             'end': 1666726701.5420592,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_169.state')},
                       'cmd_17': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.output'),
                                  'state': {'begin': 1666725970.3925776,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring_cmd" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/ng0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.057618141174316,
                                            'end': 1666725981.4501958,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_17.state')},
                       'cmd_170': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.output'),
                                   'state': {'begin': 1666726701.543887,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.06456971168518,
                                             'end': 1666726712.6084566,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_170.state')},
                       'cmd_171': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733802,\n'
                                             '  "timestamp_ms" : '
                                             '1666733802098,\n'
                                             '  "time" : "Tue Oct 25 23:36:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2776817664,\n'
                                             '        "io_kbytes" : 2711736,\n'
                                             '        "bw_bytes" : 396631576,\n'
                                             '        "bw" : 387335,\n'
                                             '        "iops" : 96833.595201,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 677932,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2374,\n'
                                             '          "max" : 177503,\n'
                                             '          "mean" : 2666.347233,\n'
                                             '          "stddev" : '
                                             '383.992040,\n'
                                             '          "N" : 677933\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 822,\n'
                                             '          "max" : 195737,\n'
                                             '          "mean" : '
                                             '17461.917285,\n'
                                             '          "stddev" : '
                                             '5079.205885,\n'
                                             '          "N" : 677933,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 12736,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '12864,\n'
                                             '            "30.000000" : '
                                             '12992,\n'
                                             '            "40.000000" : '
                                             '13120,\n'
                                             '            "50.000000" : '
                                             '20352,\n'
                                             '            "60.000000" : '
                                             '20608,\n'
                                             '            "70.000000" : '
                                             '20608,\n'
                                             '            "80.000000" : '
                                             '20864,\n'
                                             '            "90.000000" : '
                                             '20864,\n'
                                             '            "95.000000" : '
                                             '21632,\n'
                                             '            "99.000000" : '
                                             '33024,\n'
                                             '            "99.500000" : '
                                             '38144,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '65280,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13045,\n'
                                             '          "max" : 198572,\n'
                                             '          "mean" : '
                                             '20128.265079,\n'
                                             '          "stddev" : '
                                             '5254.394288,\n'
                                             '          "N" : 677933\n'
                                             '        },\n'
                                             '        "bw_min" : 385704,\n'
                                             '        "bw_max" : 389835,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '387489.428571,\n'
                                             '        "bw_dev" : 1265.535503,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 96426,\n'
                                             '        "iops_max" : 97458,\n'
                                             '        "iops_mean" : '
                                             '96872.285714,\n'
                                             '        "iops_stddev" : '
                                             '316.254870,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.857143,\n'
                                             '      "sys_cpu" : 41.557143,\n'
                                             '      "ctx" : 338781,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 47.810990,\n'
                                             '        "50" : 52.078527,\n'
                                             '        "100" : 0.101043,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 970953,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 11596,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 11596,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=86827: Tue '
                                             'Oct 25 23:36:42 2022\n'
                                             '  read: IOPS=96.8k, BW=378MiB/s '
                                             '(397MB/s)(2648MiB/7001msec)\n'
                                             '    slat (nsec): min=2374, '
                                             'max=177503, avg=2666.35, '
                                             'stdev=383.99\n'
                                             '    clat (nsec): min=822, '
                                             'max=195737, avg=17461.92, '
                                             'stdev=5079.21\n'
                                             '     lat (usec): min=13, '
                                             'max=198, avg=20.13, stdev= 5.25\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[12736], 10.00th=[12864], '
                                             '20.00th=[12864],\n'
                                             '     | 30.00th=[12992], '
                                             '40.00th=[13120], '
                                             '50.00th=[20352], '
                                             '60.00th=[20608],\n'
                                             '     | 70.00th=[20608], '
                                             '80.00th=[20864], '
                                             '90.00th=[20864], '
                                             '95.00th=[21632],\n'
                                             '     | 99.00th=[33024], '
                                             '99.50th=[38144], '
                                             '99.90th=[50944], '
                                             '99.95th=[65280],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=385704, '
                                             'max=389835, per=100.00%, '
                                             'avg=387489.43, stdev=1265.54, '
                                             'samples=14\n'
                                             '   iops        : min=96426, '
                                             'max=97458, avg=96872.29, '
                                             'stdev=316.25, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=47.81%, 50=52.08%, 100=0.10%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=14.86%, '
                                             'sys=41.56%, ctx=338781, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=677932,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=378MiB/s (397MB/s), '
                                             '378MiB/s-378MiB/s '
                                             '(397MB/s-397MB/s), io=2648MiB '
                                             '(2777MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=970953/0, '
                                             'merge=0/0, ticks=11596/0, '
                                             'in_queue=11596, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.output'),
                                   'state': {'begin': 1666726712.611358,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7903308868408203,
                                             'end': 1666726713.4016888,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_171.state')},
                       'cmd_172': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.output'),
                                   'state': {'begin': 1666726715.4345553,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7796828746795654,
                                             'end': 1666726716.2142382,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_172.state')},
                       'cmd_173': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.output'),
                                   'state': {'begin': 1666726716.2163448,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.510746002197266,
                                             'end': 1666726727.7270908,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_173.state')},
                       'cmd_174': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733817,\n'
                                             '  "timestamp_ms" : '
                                             '1666733817120,\n'
                                             '  "time" : "Tue Oct 25 23:36:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3817984000,\n'
                                             '        "io_kbytes" : 3728500,\n'
                                             '        "bw_bytes" : 545348378,\n'
                                             '        "bw" : 532566,\n'
                                             '        "iops" : 133141.551207,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 932124,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1503,\n'
                                             '          "max" : 15279,\n'
                                             '          "mean" : 1580.962068,\n'
                                             '          "stddev" : '
                                             '125.314578,\n'
                                             '          "N" : 932125\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 261,\n'
                                             '          "max" : 116468,\n'
                                             '          "mean" : '
                                             '12250.310121,\n'
                                             '          "stddev" : '
                                             '3714.120484,\n'
                                             '          "N" : 932125,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10560,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '10560,\n'
                                             '            "20.000000" : '
                                             '10688,\n'
                                             '            "30.000000" : '
                                             '10688,\n'
                                             '            "40.000000" : '
                                             '10816,\n'
                                             '            "50.000000" : '
                                             '12224,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '12352,\n'
                                             '            "80.000000" : '
                                             '12480,\n'
                                             '            "90.000000" : '
                                             '12608,\n'
                                             '            "95.000000" : '
                                             '13120,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '45312,\n'
                                             '            "99.950000" : '
                                             '63232,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9247,\n'
                                             '          "max" : 118081,\n'
                                             '          "mean" : '
                                             '13831.272190,\n'
                                             '          "stddev" : '
                                             '3720.542513,\n'
                                             '          "N" : 932125\n'
                                             '        },\n'
                                             '        "bw_min" : 531104,\n'
                                             '        "bw_max" : 534640,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '532689.428571,\n'
                                             '        "bw_dev" : 1101.338606,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 132776,\n'
                                             '        "iops_max" : 133660,\n'
                                             '        "iops_mean" : '
                                             '133172.214286,\n'
                                             '        "iops_stddev" : '
                                             '275.363306,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 84.400000,\n'
                                             '      "sys_cpu" : 15.585714,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.981869,\n'
                                             '        "2" : 50.018131,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.049886,\n'
                                             '        "20" : 96.550674,\n'
                                             '        "50" : 3.300312,\n'
                                             '        "100" : 0.096232,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=86875: Tue '
                                             'Oct 25 23:36:57 2022\n'
                                             '  read: IOPS=133k, BW=520MiB/s '
                                             '(545MB/s)(3641MiB/7001msec)\n'
                                             '    slat (nsec): min=1503, '
                                             'max=15279, avg=1580.96, '
                                             'stdev=125.31\n'
                                             '    clat (nsec): min=261, '
                                             'max=116468, avg=12250.31, '
                                             'stdev=3714.12\n'
                                             '     lat (usec): min=9, max=118, '
                                             'avg=13.83, stdev= 3.72\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10560],  '
                                             '5.00th=[10560], 10.00th=[10560], '
                                             '20.00th=[10688],\n'
                                             '     | 30.00th=[10688], '
                                             '40.00th=[10816], '
                                             '50.00th=[12224], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[12352], '
                                             '80.00th=[12480], '
                                             '90.00th=[12608], '
                                             '95.00th=[13120],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[37120], '
                                             '99.90th=[45312], '
                                             '99.95th=[63232],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=531104, '
                                             'max=534640, per=100.00%, '
                                             'avg=532689.43, stdev=1101.34, '
                                             'samples=14\n'
                                             '   iops        : min=132776, '
                                             'max=133660, avg=133172.21, '
                                             'stdev=275.36, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.05%, 20=96.55%, 50=3.30%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=84.40%, '
                                             'sys=15.59%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=932124,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=520MiB/s (545MB/s), '
                                             '520MiB/s-520MiB/s '
                                             '(545MB/s-545MB/s), io=3641MiB '
                                             '(3818MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.output'),
                                   'state': {'begin': 1666726727.7298813,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7981071472167969,
                                             'end': 1666726728.5279884,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_174.state')},
                       'cmd_175': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.output'),
                                   'state': {'begin': 1666726730.5578527,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7900984287261963,
                                             'end': 1666726731.3479512,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_175.state')},
                       'cmd_176': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.output'),
                                   'state': {'begin': 1666726731.3500626,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.429170370101929,
                                             'end': 1666726742.779233,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_176.state')},
                       'cmd_177': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733832,\n'
                                             '  "timestamp_ms" : '
                                             '1666733832158,\n'
                                             '  "time" : "Tue Oct 25 23:37:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3812896768,\n'
                                             '        "io_kbytes" : 3723532,\n'
                                             '        "bw_bytes" : 544621735,\n'
                                             '        "bw" : 531857,\n'
                                             '        "iops" : 132964.147979,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 930882,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1542,\n'
                                             '          "max" : 43872,\n'
                                             '          "mean" : 1620.137457,\n'
                                             '          "stddev" : '
                                             '131.643678,\n'
                                             '          "N" : 930882\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 240,\n'
                                             '          "max" : 1034130,\n'
                                             '          "mean" : '
                                             '12198.356105,\n'
                                             '          "stddev" : '
                                             '4029.463553,\n'
                                             '          "N" : 930882,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10432,\n'
                                             '            "5.000000" : 10432,\n'
                                             '            "10.000000" : '
                                             '10560,\n'
                                             '            "20.000000" : '
                                             '10560,\n'
                                             '            "30.000000" : '
                                             '10688,\n'
                                             '            "40.000000" : '
                                             '10688,\n'
                                             '            "50.000000" : '
                                             '12224,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '12352,\n'
                                             '            "80.000000" : '
                                             '12480,\n'
                                             '            "90.000000" : '
                                             '12480,\n'
                                             '            "95.000000" : '
                                             '13120,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9217,\n'
                                             '          "max" : 1035763,\n'
                                             '          "mean" : '
                                             '13818.493562,\n'
                                             '          "stddev" : '
                                             '4036.311352,\n'
                                             '          "N" : 930882\n'
                                             '        },\n'
                                             '        "bw_min" : 528936,\n'
                                             '        "bw_max" : 533304,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '531980.714286,\n'
                                             '        "bw_dev" : 1458.161879,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 132234,\n'
                                             '        "iops_max" : 133326,\n'
                                             '        "iops_mean" : '
                                             '132995.142857,\n'
                                             '        "iops_stddev" : '
                                             '364.505747,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 86.657143,\n'
                                             '      "sys_cpu" : 13.300000,\n'
                                             '      "ctx" : 41,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.984746,\n'
                                             '        "2" : 50.015254,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.049738,\n'
                                             '        "20" : 96.555847,\n'
                                             '        "50" : 3.289031,\n'
                                             '        "100" : 0.102483,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=86924: Tue '
                                             'Oct 25 23:37:12 2022\n'
                                             '  read: IOPS=133k, BW=519MiB/s '
                                             '(545MB/s)(3636MiB/7001msec)\n'
                                             '    slat (nsec): min=1542, '
                                             'max=43872, avg=1620.14, '
                                             'stdev=131.64\n'
                                             '    clat (nsec): min=240, '
                                             'max=1034.1k, avg=12198.36, '
                                             'stdev=4029.46\n'
                                             '     lat (usec): min=9, '
                                             'max=1035, avg=13.82, stdev= '
                                             '4.04\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10432],  '
                                             '5.00th=[10432], 10.00th=[10560], '
                                             '20.00th=[10560],\n'
                                             '     | 30.00th=[10688], '
                                             '40.00th=[10688], '
                                             '50.00th=[12224], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[12352], '
                                             '80.00th=[12480], '
                                             '90.00th=[12480], '
                                             '95.00th=[13120],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[37120], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=528936, '
                                             'max=533304, per=100.00%, '
                                             'avg=531980.71, stdev=1458.16, '
                                             'samples=14\n'
                                             '   iops        : min=132234, '
                                             'max=133326, avg=132995.14, '
                                             'stdev=364.51, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.05%, 20=96.56%, 50=3.29%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=86.66%, '
                                             'sys=13.30%, ctx=41, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=930882,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=519MiB/s (545MB/s), '
                                             '519MiB/s-519MiB/s '
                                             '(545MB/s-545MB/s), io=3636MiB '
                                             '(3813MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.output'),
                                   'state': {'begin': 1666726742.781971,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7942378520965576,
                                             'end': 1666726743.5762088,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_177.state')},
                       'cmd_178': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.output'),
                                   'state': {'begin': 1666726745.5864446,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7919878959655762,
                                             'end': 1666726746.3784325,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_178.state')},
                       'cmd_179': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.output'),
                                   'state': {'begin': 1666726746.380755,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.52251935005188,
                                             'end': 1666726757.9032743,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_179.state')},
                       'cmd_18': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733070,\n'
                                            '  "timestamp_ms" : '
                                            '1666733070951,\n'
                                            '  "time" : "Tue Oct 25 23:24:30 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring_cmd",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/ng0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1847742464,\n'
                                            '        "io_kbytes" : 1804436,\n'
                                            '        "bw_bytes" : 263925505,\n'
                                            '        "bw" : 257739,\n'
                                            '        "iops" : 64434.937866,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 451109,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2314,\n'
                                            '          "max" : 18234,\n'
                                            '          "mean" : 2506.600283,\n'
                                            '          "stddev" : 209.404360,\n'
                                            '          "N" : 451109\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 341,\n'
                                            '          "max" : 183043,\n'
                                            '          "mean" : 12344.901031,\n'
                                            '          "stddev" : '
                                            '2052.638325,\n'
                                            '          "N" : 451109,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11840,\n'
                                            '            "5.000000" : 11840,\n'
                                            '            "10.000000" : 11840,\n'
                                            '            "20.000000" : 11968,\n'
                                            '            "30.000000" : 11968,\n'
                                            '            "40.000000" : 11968,\n'
                                            '            "50.000000" : 12096,\n'
                                            '            "60.000000" : 12096,\n'
                                            '            "70.000000" : 12224,\n'
                                            '            "80.000000" : 12224,\n'
                                            '            "90.000000" : 12352,\n'
                                            '            "95.000000" : 12480,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 24448,\n'
                                            '            "99.900000" : 31616,\n'
                                            '            "99.950000" : 49920,\n'
                                            '            "99.990000" : 64256\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 11482,\n'
                                            '          "max" : 185648,\n'
                                            '          "mean" : 14851.501313,\n'
                                            '          "stddev" : '
                                            '2066.668313,\n'
                                            '          "N" : 451109\n'
                                            '        },\n'
                                            '        "bw_min" : 257312,\n'
                                            '        "bw_max" : 258252,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '257799.142857,\n'
                                            '        "bw_dev" : 320.139118,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 64328,\n'
                                            '        "iops_max" : 64563,\n'
                                            '        "iops_mean" : '
                                            '64449.785714,\n'
                                            '        "iops_stddev" : '
                                            '80.034780,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.671429,\n'
                                            '      "sys_cpu" : 36.157143,\n'
                                            '      "ctx" : 451088,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.461569,\n'
                                            '        "50" : 1.477913,\n'
                                            '        "100" : 0.051650,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=84584: Tue '
                                            'Oct 25 23:24:30 2022\n'
                                            '  read: IOPS=64.4k, BW=252MiB/s '
                                            '(264MB/s)(1762MiB/7001msec)\n'
                                            '    slat (nsec): min=2314, '
                                            'max=18234, avg=2506.60, '
                                            'stdev=209.40\n'
                                            '    clat (nsec): min=341, '
                                            'max=183043, avg=12344.90, '
                                            'stdev=2052.64\n'
                                            '     lat (usec): min=11, max=185, '
                                            'avg=14.85, stdev= 2.07\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11840],  '
                                            '5.00th=[11840], 10.00th=[11840], '
                                            '20.00th=[11968],\n'
                                            '     | 30.00th=[11968], '
                                            '40.00th=[11968], 50.00th=[12096], '
                                            '60.00th=[12096],\n'
                                            '     | 70.00th=[12224], '
                                            '80.00th=[12224], 90.00th=[12352], '
                                            '95.00th=[12480],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[24448], 99.90th=[31616], '
                                            '99.95th=[49920],\n'
                                            '     | 99.99th=[64256]\n'
                                            '   bw (  KiB/s): min=257312, '
                                            'max=258252, per=100.00%, '
                                            'avg=257799.14, stdev=320.14, '
                                            'samples=14\n'
                                            '   iops        : min=64328, '
                                            'max=64563, avg=64449.79, '
                                            'stdev=80.03, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.46%, 50=1.48%, '
                                            '100=0.05%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=10.67%, '
                                            'sys=36.16%, ctx=451088, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=451109,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=252MiB/s (264MB/s), '
                                            '252MiB/s-252MiB/s '
                                            '(264MB/s-264MB/s), io=1762MiB '
                                            '(1848MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.output'),
                                  'state': {'begin': 1666725981.4536474,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7915840148925781,
                                            'end': 1666725982.2452314,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_18.state')},
                       'cmd_180': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733847,\n'
                                             '  "timestamp_ms" : '
                                             '1666733847292,\n'
                                             '  "time" : "Tue Oct 25 23:37:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3812159488,\n'
                                             '        "io_kbytes" : 3722812,\n'
                                             '        "bw_bytes" : 544516424,\n'
                                             '        "bw" : 531754,\n'
                                             '        "iops" : 132938.437366,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 930702,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1542,\n'
                                             '          "max" : 16170,\n'
                                             '          "mean" : 1619.645983,\n'
                                             '          "stddev" : '
                                             '121.460440,\n'
                                             '          "N" : 930702\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 251,\n'
                                             '          "max" : 176772,\n'
                                             '          "mean" : '
                                             '12194.889915,\n'
                                             '          "stddev" : '
                                             '3779.824468,\n'
                                             '          "N" : 930702,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10432,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '10560,\n'
                                             '            "20.000000" : '
                                             '10560,\n'
                                             '            "30.000000" : '
                                             '10688,\n'
                                             '            "40.000000" : '
                                             '10688,\n'
                                             '            "50.000000" : '
                                             '12224,\n'
                                             '            "60.000000" : '
                                             '12352,\n'
                                             '            "70.000000" : '
                                             '12352,\n'
                                             '            "80.000000" : '
                                             '12480,\n'
                                             '            "90.000000" : '
                                             '12480,\n'
                                             '            "95.000000" : '
                                             '12992,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 74240\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9257,\n'
                                             '          "max" : 178395,\n'
                                             '          "mean" : '
                                             '13814.535899,\n'
                                             '          "stddev" : '
                                             '3787.101803,\n'
                                             '          "N" : 930702\n'
                                             '        },\n'
                                             '        "bw_min" : 530488,\n'
                                             '        "bw_max" : 532609,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '531886.928571,\n'
                                             '        "bw_dev" : 581.030446,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 132622,\n'
                                             '        "iops_max" : 133152,\n'
                                             '        "iops_mean" : '
                                             '132971.714286,\n'
                                             '        "iops_stddev" : '
                                             '145.339617,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 84.900000,\n'
                                             '      "sys_cpu" : 15.071429,\n'
                                             '      "ctx" : 16,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.984635,\n'
                                             '        "2" : 50.015365,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.048458,\n'
                                             '        "20" : 96.556255,\n'
                                             '        "50" : 3.289452,\n'
                                             '        "100" : 0.102288,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=86975: Tue '
                                             'Oct 25 23:37:27 2022\n'
                                             '  read: IOPS=133k, BW=519MiB/s '
                                             '(545MB/s)(3636MiB/7001msec)\n'
                                             '    slat (nsec): min=1542, '
                                             'max=16170, avg=1619.65, '
                                             'stdev=121.46\n'
                                             '    clat (nsec): min=251, '
                                             'max=176772, avg=12194.89, '
                                             'stdev=3779.82\n'
                                             '     lat (usec): min=9, max=178, '
                                             'avg=13.81, stdev= 3.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10432],  '
                                             '5.00th=[10560], 10.00th=[10560], '
                                             '20.00th=[10560],\n'
                                             '     | 30.00th=[10688], '
                                             '40.00th=[10688], '
                                             '50.00th=[12224], '
                                             '60.00th=[12352],\n'
                                             '     | 70.00th=[12352], '
                                             '80.00th=[12480], '
                                             '90.00th=[12480], '
                                             '95.00th=[12992],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[37120], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[74240]\n'
                                             '   bw (  KiB/s): min=530488, '
                                             'max=532609, per=100.00%, '
                                             'avg=531886.93, stdev=581.03, '
                                             'samples=14\n'
                                             '   iops        : min=132622, '
                                             'max=133152, avg=132971.71, '
                                             'stdev=145.34, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.05%, 20=96.56%, 50=3.29%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=84.90%, '
                                             'sys=15.07%, ctx=16, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=930702,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=519MiB/s (545MB/s), '
                                             '519MiB/s-519MiB/s '
                                             '(545MB/s-545MB/s), io=3636MiB '
                                             '(3812MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.output'),
                                   'state': {'begin': 1666726757.9060128,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7953801155090332,
                                             'end': 1666726758.701393,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_180.state')},
                       'cmd_181': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.output'),
                                   'state': {'begin': 1666726760.7446208,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7837724685668945,
                                             'end': 1666726761.5283933,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_181.state')},
                       'cmd_182': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.output'),
                                   'state': {'begin': 1666726761.5305362,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.477898359298706,
                                             'end': 1666726773.0084345,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_182.state')},
                       'cmd_183': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733862,\n'
                                             '  "timestamp_ms" : '
                                             '1666733862396,\n'
                                             '  "time" : "Tue Oct 25 23:37:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3758456832,\n'
                                             '        "io_kbytes" : 3670368,\n'
                                             '        "bw_bytes" : 536845712,\n'
                                             '        "bw" : 524263,\n'
                                             '        "iops" : 131065.704899,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 917591,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1542,\n'
                                             '          "max" : 1290761,\n'
                                             '          "mean" : 1616.681889,\n'
                                             '          "stddev" : '
                                             '1355.420376,\n'
                                             '          "N" : 917592\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 691,\n'
                                             '          "max" : 1291843,\n'
                                             '          "mean" : '
                                             '12424.666600,\n'
                                             '          "stddev" : '
                                             '3999.828740,\n'
                                             '          "N" : 917592,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10560,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '10688,\n'
                                             '            "20.000000" : '
                                             '10816,\n'
                                             '            "30.000000" : '
                                             '10944,\n'
                                             '            "40.000000" : '
                                             '10944,\n'
                                             '            "50.000000" : '
                                             '12352,\n'
                                             '            "60.000000" : '
                                             '12480,\n'
                                             '            "70.000000" : '
                                             '12608,\n'
                                             '            "80.000000" : '
                                             '12736,\n'
                                             '            "90.000000" : '
                                             '12736,\n'
                                             '            "95.000000" : '
                                             '13376,\n'
                                             '            "99.000000" : '
                                             '29824,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 75264\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9447,\n'
                                             '          "max" : 1294137,\n'
                                             '          "mean" : '
                                             '14041.348489,\n'
                                             '          "stddev" : '
                                             '4225.469205,\n'
                                             '          "N" : 917592\n'
                                             '        },\n'
                                             '        "bw_min" : 523344,\n'
                                             '        "bw_max" : 524897,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '524383.500000,\n'
                                             '        "bw_dev" : 486.058995,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 130836,\n'
                                             '        "iops_max" : 131224,\n'
                                             '        "iops_mean" : '
                                             '131095.857143,\n'
                                             '        "iops_stddev" : '
                                             '121.494449,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 32.271429,\n'
                                             '      "sys_cpu" : 67.685714,\n'
                                             '      "ctx" : 43,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.985451,\n'
                                             '        "2" : 50.014549,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.042394,\n'
                                             '        "20" : 96.514896,\n'
                                             '        "50" : 3.329261,\n'
                                             '        "100" : 0.110616,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87024: Tue '
                                             'Oct 25 23:37:42 2022\n'
                                             '  read: IOPS=131k, BW=512MiB/s '
                                             '(537MB/s)(3584MiB/7001msec)\n'
                                             '    slat (nsec): min=1542, '
                                             'max=1290.8k, avg=1616.68, '
                                             'stdev=1355.42\n'
                                             '    clat (nsec): min=691, '
                                             'max=1291.8k, avg=12424.67, '
                                             'stdev=3999.83\n'
                                             '     lat (usec): min=9, '
                                             'max=1294, avg=14.04, stdev= '
                                             '4.23\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10560],  '
                                             '5.00th=[10560], 10.00th=[10688], '
                                             '20.00th=[10816],\n'
                                             '     | 30.00th=[10944], '
                                             '40.00th=[10944], '
                                             '50.00th=[12352], '
                                             '60.00th=[12480],\n'
                                             '     | 70.00th=[12608], '
                                             '80.00th=[12736], '
                                             '90.00th=[12736], '
                                             '95.00th=[13376],\n'
                                             '     | 99.00th=[29824], '
                                             '99.50th=[37120], '
                                             '99.90th=[50944], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[75264]\n'
                                             '   bw (  KiB/s): min=523344, '
                                             'max=524897, per=100.00%, '
                                             'avg=524383.50, stdev=486.06, '
                                             'samples=14\n'
                                             '   iops        : min=130836, '
                                             'max=131224, avg=131095.86, '
                                             'stdev=121.49, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.04%, 20=96.51%, 50=3.33%, '
                                             '100=0.11%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=32.27%, '
                                             'sys=67.69%, ctx=43, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=917591,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=512MiB/s (537MB/s), '
                                             '512MiB/s-512MiB/s '
                                             '(537MB/s-537MB/s), io=3584MiB '
                                             '(3758MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.output'),
                                   'state': {'begin': 1666726773.0111377,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7946062088012695,
                                             'end': 1666726773.805744,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_183.state')},
                       'cmd_184': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.output'),
                                   'state': {'begin': 1666726775.8299632,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7891864776611328,
                                             'end': 1666726776.6191497,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_184.state')},
                       'cmd_185': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.output'),
                                   'state': {'begin': 1666726776.621369,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.522973537445068,
                                             'end': 1666726788.1443424,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_185.state')},
                       'cmd_186': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733877,\n'
                                             '  "timestamp_ms" : '
                                             '1666733877524,\n'
                                             '  "time" : "Tue Oct 25 23:37:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3756961792,\n'
                                             '        "io_kbytes" : 3668908,\n'
                                             '        "bw_bytes" : 536632165,\n'
                                             '        "bw" : 524054,\n'
                                             '        "iops" : 131013.569490,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 917226,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1543,\n'
                                             '          "max" : 16892,\n'
                                             '          "mean" : 1635.534015,\n'
                                             '          "stddev" : '
                                             '122.054841,\n'
                                             '          "N" : 917227\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 711,\n'
                                             '          "max" : 1299107,\n'
                                             '          "mean" : '
                                             '12389.945191,\n'
                                             '          "stddev" : '
                                             '4132.412283,\n'
                                             '          "N" : 917227,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10560,\n'
                                             '            "5.000000" : 10688,\n'
                                             '            "10.000000" : '
                                             '10688,\n'
                                             '            "20.000000" : '
                                             '10688,\n'
                                             '            "30.000000" : '
                                             '10816,\n'
                                             '            "40.000000" : '
                                             '11072,\n'
                                             '            "50.000000" : '
                                             '12352,\n'
                                             '            "60.000000" : '
                                             '12480,\n'
                                             '            "70.000000" : '
                                             '12480,\n'
                                             '            "80.000000" : '
                                             '12608,\n'
                                             '            "90.000000" : '
                                             '12864,\n'
                                             '            "95.000000" : '
                                             '13376,\n'
                                             '            "99.000000" : '
                                             '29568,\n'
                                             '            "99.500000" : '
                                             '36608,\n'
                                             '            "99.900000" : '
                                             '42240,\n'
                                             '            "99.950000" : '
                                             '62208,\n'
                                             '            "99.990000" : 75264\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9368,\n'
                                             '          "max" : 1300770,\n'
                                             '          "mean" : '
                                             '14025.479206,\n'
                                             '          "stddev" : '
                                             '4142.473008,\n'
                                             '          "N" : 917227\n'
                                             '        },\n'
                                             '        "bw_min" : 523160,\n'
                                             '        "bw_max" : 525571,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '524175.642857,\n'
                                             '        "bw_dev" : 851.873917,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 130790,\n'
                                             '        "iops_max" : 131392,\n'
                                             '        "iops_mean" : '
                                             '131043.857143,\n'
                                             '        "iops_stddev" : '
                                             '212.874053,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 35.357143,\n'
                                             '      "sys_cpu" : 64.614286,\n'
                                             '      "ctx" : 40,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.978522,\n'
                                             '        "2" : 50.021478,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.056911,\n'
                                             '        "20" : 96.554502,\n'
                                             '        "50" : 3.304638,\n'
                                             '        "100" : 0.081332,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87073: Tue '
                                             'Oct 25 23:37:57 2022\n'
                                             '  read: IOPS=131k, BW=512MiB/s '
                                             '(537MB/s)(3583MiB/7001msec)\n'
                                             '    slat (nsec): min=1543, '
                                             'max=16892, avg=1635.53, '
                                             'stdev=122.05\n'
                                             '    clat (nsec): min=711, '
                                             'max=1299.1k, avg=12389.95, '
                                             'stdev=4132.41\n'
                                             '     lat (usec): min=9, '
                                             'max=1300, avg=14.03, stdev= '
                                             '4.14\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10560],  '
                                             '5.00th=[10688], 10.00th=[10688], '
                                             '20.00th=[10688],\n'
                                             '     | 30.00th=[10816], '
                                             '40.00th=[11072], '
                                             '50.00th=[12352], '
                                             '60.00th=[12480],\n'
                                             '     | 70.00th=[12480], '
                                             '80.00th=[12608], '
                                             '90.00th=[12864], '
                                             '95.00th=[13376],\n'
                                             '     | 99.00th=[29568], '
                                             '99.50th=[36608], '
                                             '99.90th=[42240], '
                                             '99.95th=[62208],\n'
                                             '     | 99.99th=[75264]\n'
                                             '   bw (  KiB/s): min=523160, '
                                             'max=525571, per=100.00%, '
                                             'avg=524175.64, stdev=851.87, '
                                             'samples=14\n'
                                             '   iops        : min=130790, '
                                             'max=131392, avg=131043.86, '
                                             'stdev=212.87, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.06%, 20=96.55%, 50=3.30%, '
                                             '100=0.08%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=35.36%, '
                                             'sys=64.61%, ctx=40, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=917226,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=512MiB/s (537MB/s), '
                                             '512MiB/s-512MiB/s '
                                             '(537MB/s-537MB/s), io=3583MiB '
                                             '(3757MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.output'),
                                   'state': {'begin': 1666726788.146725,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7905588150024414,
                                             'end': 1666726788.9372838,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_186.state')},
                       'cmd_187': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.output'),
                                   'state': {'begin': 1666726790.9592083,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7852804660797119,
                                             'end': 1666726791.7444887,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_187.state')},
                       'cmd_188': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.output'),
                                   'state': {'begin': 1666726791.7467859,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.554159879684448,
                                             'end': 1666726803.3009458,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_188.state')},
                       'cmd_189': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733892,\n'
                                             '  "timestamp_ms" : '
                                             '1666733892681,\n'
                                             '  "time" : "Tue Oct 25 23:38:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '3749478400,\n'
                                             '        "io_kbytes" : 3661600,\n'
                                             '        "bw_bytes" : 535563262,\n'
                                             '        "bw" : 523010,\n'
                                             '        "iops" : 130752.606770,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 915399,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1552,\n'
                                             '          "max" : 175259,\n'
                                             '          "mean" : 1626.508294,\n'
                                             '          "stddev" : '
                                             '223.366531,\n'
                                             '          "N" : 915399\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 711,\n'
                                             '          "max" : 181591,\n'
                                             '          "mean" : '
                                             '12431.741996,\n'
                                             '          "stddev" : '
                                             '3780.590270,\n'
                                             '          "N" : 915399,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10560,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '10688,\n'
                                             '            "20.000000" : '
                                             '10816,\n'
                                             '            "30.000000" : '
                                             '10944,\n'
                                             '            "40.000000" : '
                                             '11072,\n'
                                             '            "50.000000" : '
                                             '12352,\n'
                                             '            "60.000000" : '
                                             '12480,\n'
                                             '            "70.000000" : '
                                             '12608,\n'
                                             '            "80.000000" : '
                                             '12736,\n'
                                             '            "90.000000" : '
                                             '12864,\n'
                                             '            "95.000000" : '
                                             '13376,\n'
                                             '            "99.000000" : '
                                             '29824,\n'
                                             '            "99.500000" : '
                                             '37120,\n'
                                             '            "99.900000" : '
                                             '50432,\n'
                                             '            "99.950000" : '
                                             '63744,\n'
                                             '            "99.990000" : 80384\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9377,\n'
                                             '          "max" : 188053,\n'
                                             '          "mean" : '
                                             '14058.250290,\n'
                                             '          "stddev" : '
                                             '3793.051475,\n'
                                             '          "N" : 915399\n'
                                             '        },\n'
                                             '        "bw_min" : 521616,\n'
                                             '        "bw_max" : 524296,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '523131.285714,\n'
                                             '        "bw_dev" : 699.172965,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 130404,\n'
                                             '        "iops_max" : 131074,\n'
                                             '        "iops_mean" : '
                                             '130782.785714,\n'
                                             '        "iops_stddev" : '
                                             '174.475116,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 34.957143,\n'
                                             '      "sys_cpu" : 65.014286,\n'
                                             '      "ctx" : 17,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 49.974055,\n'
                                             '        "2" : 50.025945,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.063470,\n'
                                             '        "20" : 96.510811,\n'
                                             '        "50" : 3.320410,\n'
                                             '        "100" : 0.101267,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87122: Tue '
                                             'Oct 25 23:38:12 2022\n'
                                             '  read: IOPS=131k, BW=511MiB/s '
                                             '(536MB/s)(3576MiB/7001msec)\n'
                                             '    slat (nsec): min=1552, '
                                             'max=175259, avg=1626.51, '
                                             'stdev=223.37\n'
                                             '    clat (nsec): min=711, '
                                             'max=181591, avg=12431.74, '
                                             'stdev=3780.59\n'
                                             '     lat (usec): min=9, max=188, '
                                             'avg=14.06, stdev= 3.79\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[10560],  '
                                             '5.00th=[10560], 10.00th=[10688], '
                                             '20.00th=[10816],\n'
                                             '     | 30.00th=[10944], '
                                             '40.00th=[11072], '
                                             '50.00th=[12352], '
                                             '60.00th=[12480],\n'
                                             '     | 70.00th=[12608], '
                                             '80.00th=[12736], '
                                             '90.00th=[12864], '
                                             '95.00th=[13376],\n'
                                             '     | 99.00th=[29824], '
                                             '99.50th=[37120], '
                                             '99.90th=[50432], '
                                             '99.95th=[63744],\n'
                                             '     | 99.99th=[80384]\n'
                                             '   bw (  KiB/s): min=521616, '
                                             'max=524296, per=100.00%, '
                                             'avg=523131.29, stdev=699.17, '
                                             'samples=14\n'
                                             '   iops        : min=130404, '
                                             'max=131074, avg=130782.79, '
                                             'stdev=174.48, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.06%, 20=96.51%, 50=3.32%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=34.96%, '
                                             'sys=65.01%, ctx=17, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=50.0%, '
                                             '2=50.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=915399,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=511MiB/s (536MB/s), '
                                             '511MiB/s-511MiB/s '
                                             '(536MB/s-536MB/s), io=3576MiB '
                                             '(3749MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.output'),
                                   'state': {'begin': 1666726803.303523,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7989928722381592,
                                             'end': 1666726804.102516,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_189.state')},
                       'cmd_19': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.output'),
                                  'state': {'begin': 1666725984.2602887,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7942605018615723,
                                            'end': 1666725985.0545492,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_19.state')},
                       'cmd_190': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.output'),
                                   'state': {'begin': 1666726805.0965261,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7852075099945068,
                                             'end': 1666726805.8817337,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_190.state')},
                       'cmd_191': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.output'),
                                   'state': {'begin': 1666726805.888838,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.097291946411133,
                                             'end': 1666726816.98613,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_191.state')},
                       'cmd_192': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733906,\n'
                                             '  "timestamp_ms" : '
                                             '1666733906474,\n'
                                             '  "time" : "Tue Oct 25 23:38:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2690433024,\n'
                                             '        "io_kbytes" : 2627376,\n'
                                             '        "bw_bytes" : 384292675,\n'
                                             '        "bw" : 375285,\n'
                                             '        "iops" : 93821.311241,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 656843,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2504,\n'
                                             '          "max" : 178896,\n'
                                             '          "mean" : 2877.458288,\n'
                                             '          "stddev" : '
                                             '468.361840,\n'
                                             '          "N" : 656843\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 951,\n'
                                             '          "max" : 1352827,\n'
                                             '          "mean" : '
                                             '17919.024456,\n'
                                             '          "stddev" : '
                                             '5851.439789,\n'
                                             '          "N" : 656844,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 12864,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '12992,\n'
                                             '            "30.000000" : '
                                             '13120,\n'
                                             '            "40.000000" : '
                                             '13120,\n'
                                             '            "50.000000" : '
                                             '21376,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '21632,\n'
                                             '            "95.000000" : '
                                             '22144,\n'
                                             '            "99.000000" : '
                                             '35072,\n'
                                             '            "99.500000" : '
                                             '40192,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '64256,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12153,\n'
                                             '          "max" : 1355933,\n'
                                             '          "mean" : '
                                             '20796.483029,\n'
                                             '          "stddev" : '
                                             '6068.884642,\n'
                                             '          "N" : 656844\n'
                                             '        },\n'
                                             '        "bw_min" : 374056,\n'
                                             '        "bw_max" : 377312,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '375377.071429,\n'
                                             '        "bw_dev" : 779.433371,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 93514,\n'
                                             '        "iops_max" : 94328,\n'
                                             '        "iops_mean" : '
                                             '93844.214286,\n'
                                             '        "iops_stddev" : '
                                             '194.825436,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.200000,\n'
                                             '      "sys_cpu" : 45.857143,\n'
                                             '      "ctx" : 328203,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 47.701962,\n'
                                             '        "50" : 52.184160,\n'
                                             '        "100" : 0.103678,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87161: Tue '
                                             'Oct 25 23:38:26 2022\n'
                                             '  read: IOPS=93.8k, BW=366MiB/s '
                                             '(384MB/s)(2566MiB/7001msec)\n'
                                             '    slat (nsec): min=2504, '
                                             'max=178896, avg=2877.46, '
                                             'stdev=468.36\n'
                                             '    clat (nsec): min=951, '
                                             'max=1352.8k, avg=17919.02, '
                                             'stdev=5851.44\n'
                                             '     lat (usec): min=12, '
                                             'max=1355, avg=20.80, stdev= '
                                             '6.07\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[12864], 10.00th=[12992], '
                                             '20.00th=[12992],\n'
                                             '     | 30.00th=[13120], '
                                             '40.00th=[13120], '
                                             '50.00th=[21376], '
                                             '60.00th=[21376],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21632], '
                                             '90.00th=[21632], '
                                             '95.00th=[22144],\n'
                                             '     | 99.00th=[35072], '
                                             '99.50th=[40192], '
                                             '99.90th=[50944], '
                                             '99.95th=[64256],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=374056, '
                                             'max=377312, per=100.00%, '
                                             'avg=375377.07, stdev=779.43, '
                                             'samples=14\n'
                                             '   iops        : min=93514, '
                                             'max=94328, avg=93844.21, '
                                             'stdev=194.83, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.01%, 20=47.70%, 50=52.18%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=15.20%, '
                                             'sys=45.86%, ctx=328203, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=656843,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=366MiB/s (384MB/s), '
                                             '366MiB/s-366MiB/s '
                                             '(384MB/s-384MB/s), io=2566MiB '
                                             '(2690MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.output'),
                                   'state': {'begin': 1666726816.9890933,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7825303077697754,
                                             'end': 1666726817.7716236,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_192.state')},
                       'cmd_193': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.output'),
                                   'state': {'begin': 1666726818.7582977,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.78106689453125,
                                             'end': 1666726819.5393646,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_193.state')},
                       'cmd_194': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.output'),
                                   'state': {'begin': 1666726819.541263,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.086409330368042,
                                             'end': 1666726830.6276724,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_194.state')},
                       'cmd_195': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733920,\n'
                                             '  "timestamp_ms" : '
                                             '1666733920115,\n'
                                             '  "time" : "Tue Oct 25 23:38:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2698158080,\n'
                                             '        "io_kbytes" : 2634920,\n'
                                             '        "bw_bytes" : 385396097,\n'
                                             '        "bw" : 376363,\n'
                                             '        "iops" : 94090.701328,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 658729,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2534,\n'
                                             '          "max" : 47179,\n'
                                             '          "mean" : 2900.782150,\n'
                                             '          "stddev" : '
                                             '418.117679,\n'
                                             '          "N" : 658729\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 921,\n'
                                             '          "max" : 118352,\n'
                                             '          "mean" : '
                                             '17829.021652,\n'
                                             '          "stddev" : '
                                             '5403.400026,\n'
                                             '          "N" : 658730,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12608,\n'
                                             '            "5.000000" : 12736,\n'
                                             '            "10.000000" : '
                                             '12736,\n'
                                             '            "20.000000" : '
                                             '12864,\n'
                                             '            "30.000000" : '
                                             '12864,\n'
                                             '            "40.000000" : '
                                             '12992,\n'
                                             '            "50.000000" : '
                                             '21376,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '21632,\n'
                                             '            "95.000000" : '
                                             '22144,\n'
                                             '            "99.000000" : '
                                             '34560,\n'
                                             '            "99.500000" : '
                                             '39680,\n'
                                             '            "99.900000" : '
                                             '48896,\n'
                                             '            "99.950000" : '
                                             '61184,\n'
                                             '            "99.990000" : 76288\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12253,\n'
                                             '          "max" : 121407,\n'
                                             '          "mean" : '
                                             '20729.804006,\n'
                                             '          "stddev" : '
                                             '5628.504721,\n'
                                             '          "N" : 658730\n'
                                             '        },\n'
                                             '        "bw_min" : 375336,\n'
                                             '        "bw_max" : 377432,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '376463.000000,\n'
                                             '        "bw_dev" : 622.233570,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 93834,\n'
                                             '        "iops_max" : 94358,\n'
                                             '        "iops_mean" : '
                                             '94115.714286,\n'
                                             '        "iops_stddev" : '
                                             '155.503860,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.242857,\n'
                                             '      "sys_cpu" : 46.571429,\n'
                                             '      "ctx" : 329136,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010019,\n'
                                             '        "20" : 47.720686,\n'
                                             '        "50" : 52.176995,\n'
                                             '        "100" : 0.090629,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87201: Tue '
                                             'Oct 25 23:38:40 2022\n'
                                             '  read: IOPS=94.1k, BW=368MiB/s '
                                             '(385MB/s)(2573MiB/7001msec)\n'
                                             '    slat (nsec): min=2534, '
                                             'max=47179, avg=2900.78, '
                                             'stdev=418.12\n'
                                             '    clat (nsec): min=921, '
                                             'max=118352, avg=17829.02, '
                                             'stdev=5403.40\n'
                                             '     lat (usec): min=12, '
                                             'max=121, avg=20.73, stdev= 5.63\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12608],  '
                                             '5.00th=[12736], 10.00th=[12736], '
                                             '20.00th=[12864],\n'
                                             '     | 30.00th=[12864], '
                                             '40.00th=[12992], '
                                             '50.00th=[21376], '
                                             '60.00th=[21376],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21632], '
                                             '90.00th=[21632], '
                                             '95.00th=[22144],\n'
                                             '     | 99.00th=[34560], '
                                             '99.50th=[39680], '
                                             '99.90th=[48896], '
                                             '99.95th=[61184],\n'
                                             '     | 99.99th=[76288]\n'
                                             '   bw (  KiB/s): min=375336, '
                                             'max=377432, per=100.00%, '
                                             'avg=376463.00, stdev=622.23, '
                                             'samples=14\n'
                                             '   iops        : min=93834, '
                                             'max=94358, avg=94115.71, '
                                             'stdev=155.50, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.01%, 20=47.72%, '
                                             '50=52.18%\n'
                                             '  lat (usec)   : 100=0.09%, '
                                             '250=0.01%\n'
                                             '  cpu          : usr=15.24%, '
                                             'sys=46.57%, ctx=329136, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=658729,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=368MiB/s (385MB/s), '
                                             '368MiB/s-368MiB/s '
                                             '(385MB/s-385MB/s), io=2573MiB '
                                             '(2698MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.output'),
                                   'state': {'begin': 1666726830.6348295,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7972743511199951,
                                             'end': 1666726831.4321039,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_195.state')},
                       'cmd_196': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_2_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.output'),
                                   'state': {'begin': 1666726832.428195,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7885875701904297,
                                             'end': 1666726833.2167826,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_196.state')},
                       'cmd_197': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.output'),
                                   'state': {'begin': 1666726833.2185743,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="2" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_2_libaio-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.09030032157898,
                                             'end': 1666726844.3088746,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_197.state')},
                       'cmd_198': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=2\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733933,\n'
                                             '  "timestamp_ms" : '
                                             '1666733933796,\n'
                                             '  "time" : "Tue Oct 25 23:38:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "2",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '2710253568,\n'
                                             '        "io_kbytes" : 2646732,\n'
                                             '        "bw_bytes" : 387123777,\n'
                                             '        "bw" : 378050,\n'
                                             '        "iops" : 94512.498215,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 661682,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2494,\n'
                                             '          "max" : 179777,\n'
                                             '          "mean" : 2858.011643,\n'
                                             '          "stddev" : '
                                             '548.309512,\n'
                                             '          "N" : 661682\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 902,\n'
                                             '          "max" : 195327,\n'
                                             '          "mean" : '
                                             '17806.183939,\n'
                                             '          "stddev" : '
                                             '5345.044073,\n'
                                             '          "N" : 661683,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 12736,\n'
                                             '            "5.000000" : 12864,\n'
                                             '            "10.000000" : '
                                             '12864,\n'
                                             '            "20.000000" : '
                                             '12992,\n'
                                             '            "30.000000" : '
                                             '12992,\n'
                                             '            "40.000000" : '
                                             '13120,\n'
                                             '            "50.000000" : '
                                             '21120,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '21376,\n'
                                             '            "95.000000" : '
                                             '21888,\n'
                                             '            "99.000000" : '
                                             '35072,\n'
                                             '            "99.500000" : '
                                             '40192,\n'
                                             '            "99.900000" : '
                                             '50944,\n'
                                             '            "99.950000" : '
                                             '64768,\n'
                                             '            "99.990000" : 77312\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12413,\n'
                                             '          "max" : 201298,\n'
                                             '          "mean" : '
                                             '20664.195956,\n'
                                             '          "stddev" : '
                                             '5573.558208,\n'
                                             '          "N" : 661683\n'
                                             '        },\n'
                                             '        "bw_min" : 377240,\n'
                                             '        "bw_max" : 379552,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '378145.928571,\n'
                                             '        "bw_dev" : 652.061638,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 94310,\n'
                                             '        "iops_max" : 94888,\n'
                                             '        "iops_mean" : '
                                             '94536.428571,\n'
                                             '        "iops_stddev" : '
                                             '163.055778,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.442857,\n'
                                             '      "sys_cpu" : 43.571429,\n'
                                             '      "ctx" : 330603,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 100.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 47.726551,\n'
                                             '        "50" : 52.157683,\n'
                                             '        "100" : 0.103373,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 2,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87241: Tue '
                                             'Oct 25 23:38:53 2022\n'
                                             '  read: IOPS=94.5k, BW=369MiB/s '
                                             '(387MB/s)(2585MiB/7001msec)\n'
                                             '    slat (nsec): min=2494, '
                                             'max=179777, avg=2858.01, '
                                             'stdev=548.31\n'
                                             '    clat (nsec): min=902, '
                                             'max=195327, avg=17806.18, '
                                             'stdev=5345.04\n'
                                             '     lat (usec): min=12, '
                                             'max=201, avg=20.66, stdev= 5.57\n'
                                             '    clat percentiles (nsec):\n'
                                             '     |  1.00th=[12736],  '
                                             '5.00th=[12864], 10.00th=[12864], '
                                             '20.00th=[12992],\n'
                                             '     | 30.00th=[12992], '
                                             '40.00th=[13120], '
                                             '50.00th=[21120], '
                                             '60.00th=[21376],\n'
                                             '     | 70.00th=[21376], '
                                             '80.00th=[21376], '
                                             '90.00th=[21376], '
                                             '95.00th=[21888],\n'
                                             '     | 99.00th=[35072], '
                                             '99.50th=[40192], '
                                             '99.90th=[50944], '
                                             '99.95th=[64768],\n'
                                             '     | 99.99th=[77312]\n'
                                             '   bw (  KiB/s): min=377240, '
                                             'max=379552, per=100.00%, '
                                             'avg=378145.93, stdev=652.06, '
                                             'samples=14\n'
                                             '   iops        : min=94310, '
                                             'max=94888, avg=94536.43, '
                                             'stdev=163.06, samples=14\n'
                                             '  lat (nsec)   : 1000=0.01%\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.01%, 20=47.73%, 50=52.16%, '
                                             '100=0.10%\n'
                                             '  lat (usec)   : 250=0.01%\n'
                                             '  cpu          : usr=17.44%, '
                                             'sys=43.57%, ctx=330603, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, '
                                             '2=100.0%, 4=0.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=661682,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=2\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=369MiB/s (387MB/s), '
                                             '369MiB/s-369MiB/s '
                                             '(387MB/s-387MB/s), io=2585MiB '
                                             '(2710MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.output'),
                                   'state': {'begin': 1666726844.3117392,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_2_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.794931173324585,
                                             'end': 1666726845.1066704,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_198.state')},
                       'cmd_199': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.output'),
                                   'state': {'begin': 1666726846.1032386,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7885692119598389,
                                             'end': 1666726846.8918078,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_199.state')},
                       'cmd_20': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.output'),
                                  'state': {'begin': 1666725985.05676,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.475365400314331,
                                            'end': 1666725996.5321255,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_20.state')},
                       'cmd_200': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.output'),
                                   'state': {'begin': 1666726846.894031,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.049890756607056,
                                             'end': 1666726857.9439218,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_200.state')},
                       'cmd_201': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733947,\n'
                                             '  "timestamp_ms" : '
                                             '1666733947431,\n'
                                             '  "time" : "Tue Oct 25 23:39:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4239355904,\n'
                                             '        "io_kbytes" : 4139996,\n'
                                             '        "bw_bytes" : 605535766,\n'
                                             '        "bw" : 591343,\n'
                                             '        "iops" : 147835.309242,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1034995,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1863,\n'
                                             '          "max" : 68268,\n'
                                             '          "mean" : 2134.228236,\n'
                                             '          "stddev" : '
                                             '423.418030,\n'
                                             '          "N" : 1034996\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 210,\n'
                                             '          "max" : 170530,\n'
                                             '          "mean" : '
                                             '24422.255110,\n'
                                             '          "stddev" : '
                                             '7557.409280,\n'
                                             '          "N" : 1034998,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14528,\n'
                                             '            "5.000000" : 17024,\n'
                                             '            "10.000000" : '
                                             '17024,\n'
                                             '            "20.000000" : '
                                             '17792,\n'
                                             '            "30.000000" : '
                                             '18048,\n'
                                             '            "40.000000" : '
                                             '18560,\n'
                                             '            "50.000000" : '
                                             '28032,\n'
                                             '            "60.000000" : '
                                             '28032,\n'
                                             '            "70.000000" : '
                                             '28288,\n'
                                             '            "80.000000" : '
                                             '28800,\n'
                                             '            "90.000000" : '
                                             '29056,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '53504,\n'
                                             '            "99.900000" : '
                                             '82432,\n'
                                             '            "99.950000" : '
                                             '97792,\n'
                                             '            "99.990000" : '
                                             '112128\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13415,\n'
                                             '          "max" : 172494,\n'
                                             '          "mean" : '
                                             '26556.483509,\n'
                                             '          "stddev" : '
                                             '7660.333802,\n'
                                             '          "N" : 1034998\n'
                                             '        },\n'
                                             '        "bw_min" : 590184,\n'
                                             '        "bw_max" : 592416,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '591485.071429,\n'
                                             '        "bw_dev" : 642.692948,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 147546,\n'
                                             '        "iops_max" : 148104,\n'
                                             '        "iops_mean" : '
                                             '147871.214286,\n'
                                             '        "iops_stddev" : '
                                             '160.657389,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.742857,\n'
                                             '      "sys_cpu" : 57.785714,\n'
                                             '      "ctx" : 266084,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 44.522920,\n'
                                             '        "50" : 53.856106,\n'
                                             '        "100" : 1.573244,\n'
                                             '        "250" : 0.045604,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87281: Tue Oct 25 23:39:07 '
                                             '2022\n'
                                             '  read: IOPS=148k, BW=577MiB/s '
                                             '(606MB/s)(4043MiB/7001msec)\n'
                                             '    slat (nsec): min=1863, '
                                             'max=68268, avg=2134.23, '
                                             'stdev=423.42\n'
                                             '    clat (nsec): min=210, '
                                             'max=170530, avg=24422.26, '
                                             'stdev=7557.41\n'
                                             '     lat (usec): min=13, '
                                             'max=172, avg=26.56, stdev= 7.66\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   18], 10.00th=[   18], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   19], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   29], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   54], 99.90th=[   '
                                             '83], 99.95th=[   98],\n'
                                             '     | 99.99th=[  113]\n'
                                             '   bw (  KiB/s): min=590184, '
                                             'max=592416, per=100.00%, '
                                             'avg=591485.07, stdev=642.69, '
                                             'samples=14\n'
                                             '   iops        : min=147546, '
                                             'max=148104, avg=147871.21, '
                                             'stdev=160.66, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=44.52%, 50=53.86%, '
                                             '100=1.57%\n'
                                             '  lat (usec)   : 250=0.05%\n'
                                             '  cpu          : usr=14.74%, '
                                             'sys=57.79%, ctx=266084, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1034995,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=577MiB/s (606MB/s), '
                                             '577MiB/s-577MiB/s '
                                             '(606MB/s-606MB/s), io=4043MiB '
                                             '(4239MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.output'),
                                   'state': {'begin': 1666726857.9465978,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7997150421142578,
                                             'end': 1666726858.7463129,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_201.state')},
                       'cmd_202': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.output'),
                                   'state': {'begin': 1666726859.737454,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.794926643371582,
                                             'end': 1666726860.5323806,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_202.state')},
                       'cmd_203': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.output'),
                                   'state': {'begin': 1666726860.5341961,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.052112102508545,
                                             'end': 1666726871.5863082,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_203.state')},
                       'cmd_204': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733961,\n'
                                             '  "timestamp_ms" : '
                                             '1666733961073,\n'
                                             '  "time" : "Tue Oct 25 23:39:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4248465408,\n'
                                             '        "io_kbytes" : 4148892,\n'
                                             '        "bw_bytes" : 606836938,\n'
                                             '        "bw" : 592614,\n'
                                             '        "iops" : 148152.978146,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1037219,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1874,\n'
                                             '          "max" : 27602,\n'
                                             '          "mean" : 2143.573774,\n'
                                             '          "stddev" : '
                                             '413.583393,\n'
                                             '          "N" : 1037220\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2985,\n'
                                             '          "max" : 204163,\n'
                                             '          "mean" : '
                                             '24353.154686,\n'
                                             '          "stddev" : '
                                             '7561.186398,\n'
                                             '          "N" : 1037222,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16768,\n'
                                             '            "5.000000" : 17024,\n'
                                             '            "10.000000" : '
                                             '17024,\n'
                                             '            "20.000000" : '
                                             '17792,\n'
                                             '            "30.000000" : '
                                             '17792,\n'
                                             '            "40.000000" : '
                                             '18560,\n'
                                             '            "50.000000" : '
                                             '28032,\n'
                                             '            "60.000000" : '
                                             '28032,\n'
                                             '            "70.000000" : '
                                             '28288,\n'
                                             '            "80.000000" : '
                                             '28800,\n'
                                             '            "90.000000" : '
                                             '29056,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '53504,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '96768,\n'
                                             '            "99.990000" : '
                                             '113152\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13285,\n'
                                             '          "max" : 206557,\n'
                                             '          "mean" : '
                                             '26496.728306,\n'
                                             '          "stddev" : '
                                             '7666.456687,\n'
                                             '          "N" : 1037222\n'
                                             '        },\n'
                                             '        "bw_min" : 591632,\n'
                                             '        "bw_max" : 593816,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '592752.785714,\n'
                                             '        "bw_dev" : 705.015022,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 147908,\n'
                                             '        "iops_max" : 148454,\n'
                                             '        "iops_mean" : '
                                             '148188.142857,\n'
                                             '        "iops_stddev" : '
                                             '176.173803,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.328571,\n'
                                             '      "sys_cpu" : 55.942857,\n'
                                             '      "ctx" : 266741,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 45.803249,\n'
                                             '        "50" : 52.586484,\n'
                                             '        "100" : 1.563893,\n'
                                             '        "250" : 0.044157,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87319: Tue Oct 25 23:39:21 '
                                             '2022\n'
                                             '  read: IOPS=148k, BW=579MiB/s '
                                             '(607MB/s)(4052MiB/7001msec)\n'
                                             '    slat (nsec): min=1874, '
                                             'max=27602, avg=2143.57, '
                                             'stdev=413.58\n'
                                             '    clat (usec): min=2, max=204, '
                                             'avg=24.35, stdev= 7.56\n'
                                             '     lat (usec): min=13, '
                                             'max=206, avg=26.50, stdev= 7.67\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   17],  '
                                             '5.00th=[   18], 10.00th=[   18], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   29], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   54], 99.90th=[   '
                                             '82], 99.95th=[   97],\n'
                                             '     | 99.99th=[  114]\n'
                                             '   bw (  KiB/s): min=591632, '
                                             'max=593816, per=100.00%, '
                                             'avg=592752.79, stdev=705.02, '
                                             'samples=14\n'
                                             '   iops        : min=147908, '
                                             'max=148454, avg=148188.14, '
                                             'stdev=176.17, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=45.80%, 50=52.59%, '
                                             '100=1.56%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=16.33%, '
                                             'sys=55.94%, ctx=266741, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1037219,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=579MiB/s (607MB/s), '
                                             '579MiB/s-579MiB/s '
                                             '(607MB/s-607MB/s), io=4052MiB '
                                             '(4248MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.output'),
                                   'state': {'begin': 1666726871.5889437,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.793954610824585,
                                             'end': 1666726872.3828983,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_204.state')},
                       'cmd_205': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.output'),
                                   'state': {'begin': 1666726873.4409869,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7898757457733154,
                                             'end': 1666726874.2308626,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_205.state')},
                       'cmd_206': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.output'),
                                   'state': {'begin': 1666726874.2328246,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.050575256347656,
                                             'end': 1666726885.2833998,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_206.state')},
                       'cmd_207': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733974,\n'
                                             '  "timestamp_ms" : '
                                             '1666733974770,\n'
                                             '  "time" : "Tue Oct 25 23:39:34 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4263362560,\n'
                                             '        "io_kbytes" : 4163440,\n'
                                             '        "bw_bytes" : 608964799,\n'
                                             '        "bw" : 594692,\n'
                                             '        "iops" : 148672.618197,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1040857,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1883,\n'
                                             '          "max" : 30347,\n'
                                             '          "mean" : 2119.325331,\n'
                                             '          "stddev" : '
                                             '420.060474,\n'
                                             '          "N" : 1040857\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 251,\n'
                                             '          "max" : 149521,\n'
                                             '          "mean" : '
                                             '24289.765743,\n'
                                             '          "stddev" : '
                                             '7494.997171,\n'
                                             '          "N" : 1040860,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15168,\n'
                                             '            "5.000000" : 17024,\n'
                                             '            "10.000000" : '
                                             '17024,\n'
                                             '            "20.000000" : '
                                             '17792,\n'
                                             '            "30.000000" : '
                                             '17792,\n'
                                             '            "40.000000" : '
                                             '18560,\n'
                                             '            "50.000000" : '
                                             '27776,\n'
                                             '            "60.000000" : '
                                             '28032,\n'
                                             '            "70.000000" : '
                                             '28288,\n'
                                             '            "80.000000" : '
                                             '28800,\n'
                                             '            "90.000000" : '
                                             '29056,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '53504,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '97792,\n'
                                             '            "99.990000" : '
                                             '113152\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14096,\n'
                                             '          "max" : 151475,\n'
                                             '          "mean" : '
                                             '26409.091097,\n'
                                             '          "stddev" : '
                                             '7588.653299,\n'
                                             '          "N" : 1040860\n'
                                             '        },\n'
                                             '        "bw_min" : 593880,\n'
                                             '        "bw_max" : 595576,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '594850.642857,\n'
                                             '        "bw_dev" : 491.431129,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 148470,\n'
                                             '        "iops_max" : 148894,\n'
                                             '        "iops_mean" : '
                                             '148712.642857,\n'
                                             '        "iops_stddev" : '
                                             '122.870543,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.671429,\n'
                                             '      "sys_cpu" : 58.242857,\n'
                                             '      "ctx" : 267603,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 45.125891,\n'
                                             '        "50" : 53.340180,\n'
                                             '        "100" : 1.487716,\n'
                                             '        "250" : 0.043714,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87357: Tue Oct 25 23:39:34 '
                                             '2022\n'
                                             '  read: IOPS=149k, BW=581MiB/s '
                                             '(609MB/s)(4066MiB/7001msec)\n'
                                             '    slat (nsec): min=1883, '
                                             'max=30347, avg=2119.33, '
                                             'stdev=420.06\n'
                                             '    clat (nsec): min=251, '
                                             'max=149521, avg=24289.77, '
                                             'stdev=7495.00\n'
                                             '     lat (usec): min=14, '
                                             'max=151, avg=26.41, stdev= 7.59\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   16],  '
                                             '5.00th=[   18], 10.00th=[   18], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '28], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   29], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   54], 99.90th=[   '
                                             '82], 99.95th=[   98],\n'
                                             '     | 99.99th=[  114]\n'
                                             '   bw (  KiB/s): min=593880, '
                                             'max=595576, per=100.00%, '
                                             'avg=594850.64, stdev=491.43, '
                                             'samples=14\n'
                                             '   iops        : min=148470, '
                                             'max=148894, avg=148712.64, '
                                             'stdev=122.87, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=45.13%, 50=53.34%, '
                                             '100=1.49%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=13.67%, '
                                             'sys=58.24%, ctx=267603, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1040857,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=581MiB/s (609MB/s), '
                                             '581MiB/s-581MiB/s '
                                             '(609MB/s-609MB/s), io=4066MiB '
                                             '(4263MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.output'),
                                   'state': {'begin': 1666726885.2859445,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7888407707214355,
                                             'end': 1666726886.0747852,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_207.state')},
                       'cmd_208': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.output'),
                                   'state': {'begin': 1666726887.0683186,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7800571918487549,
                                             'end': 1666726887.8483758,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_208.state')},
                       'cmd_209': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.output'),
                                   'state': {'begin': 1666726887.8502033,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.055967092514038,
                                             'end': 1666726898.9061704,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_209.state')},
                       'cmd_21': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733085,\n'
                                            '  "timestamp_ms" : '
                                            '1666733085963,\n'
                                            '  "time" : "Tue Oct 25 23:24:45 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2405982208,\n'
                                            '        "io_kbytes" : 2349592,\n'
                                            '        "bw_bytes" : 343662649,\n'
                                            '        "bw" : 335608,\n'
                                            '        "iops" : 83902.013998,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 587398,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 711,\n'
                                            '          "max" : 4368,\n'
                                            '          "mean" : 924.473922,\n'
                                            '          "stddev" : 69.051652,\n'
                                            '          "N" : 587398\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 9077,\n'
                                            '          "max" : 113724,\n'
                                            '          "mean" : 10706.197505,\n'
                                            '          "stddev" : '
                                            '2217.342043,\n'
                                            '          "N" : 587398,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9920,\n'
                                            '            "5.000000" : 10048,\n'
                                            '            "10.000000" : 10048,\n'
                                            '            "20.000000" : 10176,\n'
                                            '            "30.000000" : 10304,\n'
                                            '            "40.000000" : 10304,\n'
                                            '            "50.000000" : 10432,\n'
                                            '            "60.000000" : 10432,\n'
                                            '            "70.000000" : 10560,\n'
                                            '            "80.000000" : 10560,\n'
                                            '            "90.000000" : 10688,\n'
                                            '            "95.000000" : 11072,\n'
                                            '            "99.000000" : 20096,\n'
                                            '            "99.500000" : 27520,\n'
                                            '            "99.900000" : 30080,\n'
                                            '            "99.950000" : 48384,\n'
                                            '            "99.990000" : 68096\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 10510,\n'
                                            '          "max" : 114625,\n'
                                            '          "mean" : 11630.671427,\n'
                                            '          "stddev" : '
                                            '2218.545088,\n'
                                            '          "N" : 587398\n'
                                            '        },\n'
                                            '        "bw_min" : 335280,\n'
                                            '        "bw_max" : 336168,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '335695.428571,\n'
                                            '        "bw_dev" : 291.592369,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83820,\n'
                                            '        "iops_max" : 84042,\n'
                                            '        "iops_mean" : '
                                            '83923.857143,\n'
                                            '        "iops_stddev" : '
                                            '72.898092,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 16,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 3.081897,\n'
                                            '        "20" : 95.920653,\n'
                                            '        "50" : 0.950122,\n'
                                            '        "100" : 0.047157,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84633: Tue Oct 25 23:24:45 '
                                            '2022\n'
                                            '  read: IOPS=83.9k, BW=328MiB/s '
                                            '(344MB/s)(2295MiB/7001msec)\n'
                                            '    slat (nsec): min=711, '
                                            'max=4368, avg=924.47, '
                                            'stdev=69.05\n'
                                            '    clat (usec): min=9, max=113, '
                                            'avg=10.71, stdev= 2.22\n'
                                            '     lat (usec): min=10, max=114, '
                                            'avg=11.63, stdev= 2.22\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9920],  '
                                            '5.00th=[10048], 10.00th=[10048], '
                                            '20.00th=[10176],\n'
                                            '     | 30.00th=[10304], '
                                            '40.00th=[10304], 50.00th=[10432], '
                                            '60.00th=[10432],\n'
                                            '     | 70.00th=[10560], '
                                            '80.00th=[10560], 90.00th=[10688], '
                                            '95.00th=[11072],\n'
                                            '     | 99.00th=[20096], '
                                            '99.50th=[27520], 99.90th=[30080], '
                                            '99.95th=[48384],\n'
                                            '     | 99.99th=[68096]\n'
                                            '   bw (  KiB/s): min=335280, '
                                            'max=336168, per=100.00%, '
                                            'avg=335695.43, stdev=291.59, '
                                            'samples=14\n'
                                            '   iops        : min=83820, '
                                            'max=84042, avg=83923.86, '
                                            'stdev=72.90, samples=14\n'
                                            '  lat (usec)   : 10=3.08%, '
                                            '20=95.92%, 50=0.95%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=16, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=587398,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=328MiB/s (344MB/s), '
                                            '328MiB/s-328MiB/s '
                                            '(344MB/s-344MB/s), io=2295MiB '
                                            '(2406MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.output'),
                                  'state': {'begin': 1666725996.534893,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7965598106384277,
                                            'end': 1666725997.3314528,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_21.state')},
                       'cmd_210': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666733988,\n'
                                             '  "timestamp_ms" : '
                                             '1666733988393,\n'
                                             '  "time" : "Tue Oct 25 23:39:48 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4244062208,\n'
                                             '        "io_kbytes" : 4144592,\n'
                                             '        "bw_bytes" : 606208000,\n'
                                             '        "bw" : 592000,\n'
                                             '        "iops" : 147999.428653,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1036144,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1973,\n'
                                             '          "max" : 32401,\n'
                                             '          "mean" : 2217.550113,\n'
                                             '          "stddev" : '
                                             '422.893349,\n'
                                             '          "N" : 1036145\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 260,\n'
                                             '          "max" : 149982,\n'
                                             '          "mean" : '
                                             '24348.226217,\n'
                                             '          "stddev" : '
                                             '7711.887494,\n'
                                             '          "N" : 1036147,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17536,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '18304,\n'
                                             '            "50.000000" : '
                                             '28288,\n'
                                             '            "60.000000" : '
                                             '28288,\n'
                                             '            "70.000000" : '
                                             '28544,\n'
                                             '            "80.000000" : '
                                             '29056,\n'
                                             '            "90.000000" : '
                                             '29312,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '53504,\n'
                                             '            "99.900000" : '
                                             '80384,\n'
                                             '            "99.950000" : '
                                             '96768,\n'
                                             '            "99.990000" : '
                                             '113152\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13815,\n'
                                             '          "max" : 152456,\n'
                                             '          "mean" : '
                                             '26565.776468,\n'
                                             '          "stddev" : '
                                             '7822.726275,\n'
                                             '          "N" : 1036147\n'
                                             '        },\n'
                                             '        "bw_min" : 590720,\n'
                                             '        "bw_max" : 593480,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '592140.500000,\n'
                                             '        "bw_dev" : 670.367379,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 147680,\n'
                                             '        "iops_max" : 148370,\n'
                                             '        "iops_mean" : '
                                             '148035.071429,\n'
                                             '        "iops_stddev" : '
                                             '167.612403,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.842857,\n'
                                             '      "sys_cpu" : 56.142857,\n'
                                             '      "ctx" : 265858,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 45.727042,\n'
                                             '        "50" : 52.683121,\n'
                                             '        "100" : 1.546310,\n'
                                             '        "250" : 0.040824,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87395: Tue '
                                             'Oct 25 23:39:48 2022\n'
                                             '  read: IOPS=148k, BW=578MiB/s '
                                             '(606MB/s)(4047MiB/7001msec)\n'
                                             '    slat (nsec): min=1973, '
                                             'max=32401, avg=2217.55, '
                                             'stdev=422.89\n'
                                             '    clat (nsec): min=260, '
                                             'max=149982, avg=24348.23, '
                                             'stdev=7711.89\n'
                                             '     lat (usec): min=13, '
                                             'max=152, avg=26.57, stdev= 7.82\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   17],  '
                                             '5.00th=[   17], 10.00th=[   17], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   30], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   54], 99.90th=[   '
                                             '81], 99.95th=[   97],\n'
                                             '     | 99.99th=[  114]\n'
                                             '   bw (  KiB/s): min=590720, '
                                             'max=593480, per=100.00%, '
                                             'avg=592140.50, stdev=670.37, '
                                             'samples=14\n'
                                             '   iops        : min=147680, '
                                             'max=148370, avg=148035.07, '
                                             'stdev=167.61, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=45.73%, 50=52.68%, '
                                             '100=1.55%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=17.84%, '
                                             'sys=56.14%, ctx=265858, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1036144,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=578MiB/s (606MB/s), '
                                             '578MiB/s-578MiB/s '
                                             '(606MB/s-606MB/s), io=4047MiB '
                                             '(4244MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.output'),
                                   'state': {'begin': 1666726898.9089272,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.9105038642883301,
                                             'end': 1666726899.819431,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_210.state')},
                       'cmd_211': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.output'),
                                   'state': {'begin': 1666726900.888153,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8740639686584473,
                                             'end': 1666726901.762217,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_211.state')},
                       'cmd_212': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.output'),
                                   'state': {'begin': 1666726901.7642074,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.154089212417603,
                                             'end': 1666726912.9182966,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_212.state')},
                       'cmd_213': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734002,\n'
                                             '  "timestamp_ms" : '
                                             '1666734002404,\n'
                                             '  "time" : "Tue Oct 25 23:40:02 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4233482240,\n'
                                             '        "io_kbytes" : 4134260,\n'
                                             '        "bw_bytes" : 604696791,\n'
                                             '        "bw" : 590524,\n'
                                             '        "iops" : 147630.624197,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1033562,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1973,\n'
                                             '          "max" : 28543,\n'
                                             '          "mean" : 2227.234882,\n'
                                             '          "stddev" : '
                                             '433.965898,\n'
                                             '          "N" : 1033562\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 291,\n'
                                             '          "max" : 150883,\n'
                                             '          "mean" : '
                                             '24397.982275,\n'
                                             '          "stddev" : '
                                             '7660.972958,\n'
                                             '          "N" : 1033565,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16512,\n'
                                             '            "5.000000" : 16768,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17536,\n'
                                             '            "30.000000" : '
                                             '17792,\n'
                                             '            "40.000000" : '
                                             '18304,\n'
                                             '            "50.000000" : '
                                             '28288,\n'
                                             '            "60.000000" : '
                                             '28544,\n'
                                             '            "70.000000" : '
                                             '28544,\n'
                                             '            "80.000000" : '
                                             '29056,\n'
                                             '            "90.000000" : '
                                             '29312,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '52992,\n'
                                             '            "99.900000" : '
                                             '76288,\n'
                                             '            "99.950000" : '
                                             '94720,\n'
                                             '            "99.990000" : '
                                             '112128\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13616,\n'
                                             '          "max" : 153037,\n'
                                             '          "mean" : '
                                             '26625.217129,\n'
                                             '          "stddev" : '
                                             '7783.374398,\n'
                                             '          "N" : 1033565\n'
                                             '        },\n'
                                             '        "bw_min" : 589488,\n'
                                             '        "bw_max" : 591486,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '590663.285714,\n'
                                             '        "bw_dev" : 586.088787,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 147372,\n'
                                             '        "iops_max" : 147871,\n'
                                             '        "iops_mean" : '
                                             '147665.785714,\n'
                                             '        "iops_stddev" : '
                                             '146.583753,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.671429,\n'
                                             '      "sys_cpu" : 55.157143,\n'
                                             '      "ctx" : 265038,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 45.679117,\n'
                                             '        "50" : 52.747102,\n'
                                             '        "100" : 1.533628,\n'
                                             '        "250" : 0.037540,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87435: Tue '
                                             'Oct 25 23:40:02 2022\n'
                                             '  read: IOPS=148k, BW=577MiB/s '
                                             '(605MB/s)(4037MiB/7001msec)\n'
                                             '    slat (nsec): min=1973, '
                                             'max=28543, avg=2227.23, '
                                             'stdev=433.97\n'
                                             '    clat (nsec): min=291, '
                                             'max=150883, avg=24397.98, '
                                             'stdev=7660.97\n'
                                             '     lat (usec): min=13, '
                                             'max=153, avg=26.63, stdev= 7.78\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   17],  '
                                             '5.00th=[   17], 10.00th=[   17], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   30], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   53], 99.90th=[   '
                                             '77], 99.95th=[   95],\n'
                                             '     | 99.99th=[  113]\n'
                                             '   bw (  KiB/s): min=589488, '
                                             'max=591486, per=100.00%, '
                                             'avg=590663.29, stdev=586.09, '
                                             'samples=14\n'
                                             '   iops        : min=147372, '
                                             'max=147871, avg=147665.79, '
                                             'stdev=146.58, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=45.68%, 50=52.75%, '
                                             '100=1.53%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=18.67%, '
                                             'sys=55.16%, ctx=265038, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1033562,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=577MiB/s (605MB/s), '
                                             '577MiB/s-577MiB/s '
                                             '(605MB/s-605MB/s), io=4037MiB '
                                             '(4233MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.output'),
                                   'state': {'begin': 1666726912.9211314,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8747694492340088,
                                             'end': 1666726913.7959008,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_213.state')},
                       'cmd_214': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.output'),
                                   'state': {'begin': 1666726914.929846,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863180637359619,
                                             'end': 1666726915.716164,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_214.state')},
                       'cmd_215': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.output'),
                                   'state': {'begin': 1666726915.718231,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.057596683502197,
                                             'end': 1666726926.7758276,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_215.state')},
                       'cmd_216': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734016,\n'
                                             '  "timestamp_ms" : '
                                             '1666734016263,\n'
                                             '  "time" : "Tue Oct 25 23:40:16 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4226142208,\n'
                                             '        "io_kbytes" : 4127092,\n'
                                             '        "bw_bytes" : 603648365,\n'
                                             '        "bw" : 589500,\n'
                                             '        "iops" : 147374.517926,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1031769,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1993,\n'
                                             '          "max" : 26640,\n'
                                             '          "mean" : 2267.380921,\n'
                                             '          "stddev" : '
                                             '434.200456,\n'
                                             '          "N" : 1031770\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 310,\n'
                                             '          "max" : 200466,\n'
                                             '          "mean" : '
                                             '24390.233048,\n'
                                             '          "stddev" : '
                                             '7872.725576,\n'
                                             '          "N" : 1031772,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 16192,\n'
                                             '            "5.000000" : 16512,\n'
                                             '            "10.000000" : '
                                             '16768,\n'
                                             '            "20.000000" : '
                                             '17280,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '18304,\n'
                                             '            "50.000000" : '
                                             '28288,\n'
                                             '            "60.000000" : '
                                             '28544,\n'
                                             '            "70.000000" : '
                                             '28800,\n'
                                             '            "80.000000" : '
                                             '29056,\n'
                                             '            "90.000000" : '
                                             '29568,\n'
                                             '            "95.000000" : '
                                             '34048,\n'
                                             '            "99.000000" : '
                                             '51968,\n'
                                             '            "99.500000" : '
                                             '53504,\n'
                                             '            "99.900000" : '
                                             '84480,\n'
                                             '            "99.950000" : '
                                             '99840,\n'
                                             '            "99.990000" : '
                                             '115200\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13556,\n'
                                             '          "max" : 203111,\n'
                                             '          "mean" : '
                                             '26657.614235,\n'
                                             '          "stddev" : '
                                             '7990.434757,\n'
                                             '          "N" : 1031772\n'
                                             '        },\n'
                                             '        "bw_min" : 587911,\n'
                                             '        "bw_max" : 591096,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '589642.214286,\n'
                                             '        "bw_dev" : 928.622982,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 146977,\n'
                                             '        "iops_max" : 147774,\n'
                                             '        "iops_mean" : '
                                             '147410.500000,\n'
                                             '        "iops_stddev" : '
                                             '232.283232,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.514286,\n'
                                             '      "sys_cpu" : 56.057143,\n'
                                             '      "ctx" : 264480,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 45.311111,\n'
                                             '        "50" : 53.090178,\n'
                                             '        "100" : 1.546858,\n'
                                             '        "250" : 0.049139,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87473: Tue '
                                             'Oct 25 23:40:16 2022\n'
                                             '  read: IOPS=147k, BW=576MiB/s '
                                             '(604MB/s)(4030MiB/7001msec)\n'
                                             '    slat (nsec): min=1993, '
                                             'max=26640, avg=2267.38, '
                                             'stdev=434.20\n'
                                             '    clat (nsec): min=310, '
                                             'max=200466, avg=24390.23, '
                                             'stdev=7872.73\n'
                                             '     lat (usec): min=13, '
                                             'max=203, avg=26.66, stdev= 7.99\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   17],  '
                                             '5.00th=[   17], 10.00th=[   17], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   19], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   30], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   52], '
                                             '99.50th=[   54], 99.90th=[   '
                                             '85], 99.95th=[  100],\n'
                                             '     | 99.99th=[  116]\n'
                                             '   bw (  KiB/s): min=587911, '
                                             'max=591096, per=100.00%, '
                                             'avg=589642.21, stdev=928.62, '
                                             'samples=14\n'
                                             '   iops        : min=146977, '
                                             'max=147774, avg=147410.50, '
                                             'stdev=232.28, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=45.31%, 50=53.09%, '
                                             '100=1.55%\n'
                                             '  lat (usec)   : 250=0.05%\n'
                                             '  cpu          : usr=18.51%, '
                                             'sys=56.06%, ctx=264480, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1031769,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=576MiB/s (604MB/s), '
                                             '576MiB/s-576MiB/s '
                                             '(604MB/s-604MB/s), io=4030MiB '
                                             '(4226MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.output'),
                                   'state': {'begin': 1666726926.7787688,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7961454391479492,
                                             'end': 1666726927.5749142,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_216.state')},
                       'cmd_217': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.output'),
                                   'state': {'begin': 1666726929.6196554,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7881889343261719,
                                             'end': 1666726930.4078443,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_217.state')},
                       'cmd_218': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.output'),
                                   'state': {'begin': 1666726930.4099026,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.354801654815674,
                                             'end': 1666726941.7647042,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_218.state')},
                       'cmd_219': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734031,\n'
                                             '  "timestamp_ms" : '
                                             '1666734031180,\n'
                                             '  "time" : "Tue Oct 25 23:40:31 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5484310528,\n'
                                             '        "io_kbytes" : 5355772,\n'
                                             '        "bw_bytes" : 783361023,\n'
                                             '        "bw" : 765000,\n'
                                             '        "iops" : 191249.821454,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1338940,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 661,\n'
                                             '          "max" : 38161,\n'
                                             '          "mean" : 943.770155,\n'
                                             '          "stddev" : '
                                             '157.273061,\n'
                                             '          "N" : 1338940\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1282,\n'
                                             '          "max" : 194796,\n'
                                             '          "mean" : '
                                             '19434.049774,\n'
                                             '          "stddev" : '
                                             '6553.654058,\n'
                                             '          "N" : 1338942,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 11200,\n'
                                             '            "5.000000" : 12480,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '15936,\n'
                                             '            "30.000000" : '
                                             '16320,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '18048,\n'
                                             '            "60.000000" : '
                                             '19328,\n'
                                             '            "70.000000" : '
                                             '20352,\n'
                                             '            "80.000000" : '
                                             '22656,\n'
                                             '            "90.000000" : '
                                             '24448,\n'
                                             '            "95.000000" : '
                                             '32640,\n'
                                             '            "99.000000" : '
                                             '44288,\n'
                                             '            "99.500000" : '
                                             '47360,\n'
                                             '            "99.900000" : '
                                             '77312,\n'
                                             '            "99.950000" : '
                                             '92672,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11071,\n'
                                             '          "max" : 195707,\n'
                                             '          "mean" : '
                                             '20377.819873,\n'
                                             '          "stddev" : '
                                             '6563.731917,\n'
                                             '          "N" : 1338942\n'
                                             '        },\n'
                                             '        "bw_min" : 764144,\n'
                                             '        "bw_max" : 766368,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '765234.071429,\n'
                                             '        "bw_dev" : 634.098083,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 191036,\n'
                                             '        "iops_max" : 191592,\n'
                                             '        "iops_mean" : '
                                             '191308.357143,\n'
                                             '        "iops_stddev" : '
                                             '158.356757,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 9,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 19.636429,\n'
                                             '        "4" : 80.350725,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 65.775315,\n'
                                             '        "50" : 33.972023,\n'
                                             '        "100" : 0.223236,\n'
                                             '        "250" : 0.027634,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87522: Tue Oct 25 23:40:31 '
                                             '2022\n'
                                             '  read: IOPS=191k, BW=747MiB/s '
                                             '(783MB/s)(5230MiB/7001msec)\n'
                                             '    slat (nsec): min=661, '
                                             'max=38161, avg=943.77, '
                                             'stdev=157.27\n'
                                             '    clat (nsec): min=1282, '
                                             'max=194796, avg=19434.05, '
                                             'stdev=6553.65\n'
                                             '     lat (usec): min=11, '
                                             'max=195, avg=20.38, stdev= 6.56\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   12],  '
                                             '5.00th=[   13], 10.00th=[   13], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   17], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '19], 60.00th=[   20],\n'
                                             '     | 70.00th=[   21], '
                                             '80.00th=[   23], 90.00th=[   '
                                             '25], 95.00th=[   33],\n'
                                             '     | 99.00th=[   45], '
                                             '99.50th=[   48], 99.90th=[   '
                                             '78], 99.95th=[   93],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=764144, '
                                             'max=766368, per=100.00%, '
                                             'avg=765234.07, stdev=634.10, '
                                             'samples=14\n'
                                             '   iops        : min=191036, '
                                             'max=191592, avg=191308.36, '
                                             'stdev=158.36, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.01%, 20=65.78%, '
                                             '50=33.97%\n'
                                             '  lat (usec)   : 100=0.22%, '
                                             '250=0.03%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=9, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=19.6%, 4=80.4%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1338940,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=747MiB/s (783MB/s), '
                                             '747MiB/s-747MiB/s '
                                             '(783MB/s-783MB/s), io=5230MiB '
                                             '(5484MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.output'),
                                   'state': {'begin': 1666726941.7674084,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7939178943634033,
                                             'end': 1666726942.5613263,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_219.state')},
                       'cmd_22': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.output'),
                                  'state': {'begin': 1666725999.3979034,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7901809215545654,
                                            'end': 1666726000.1880844,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_22.state')},
                       'cmd_220': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.output'),
                                   'state': {'begin': 1666726944.5957801,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7897424697875977,
                                             'end': 1666726945.3855226,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_220.state')},
                       'cmd_221': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.output'),
                                   'state': {'begin': 1666726945.3878367,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.472328424453735,
                                             'end': 1666726956.860165,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_221.state')},
                       'cmd_222': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734046,\n'
                                             '  "timestamp_ms" : '
                                             '1666734046275,\n'
                                             '  "time" : "Tue Oct 25 23:40:46 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5460344832,\n'
                                             '        "io_kbytes" : 5332368,\n'
                                             '        "bw_bytes" : 779937842,\n'
                                             '        "bw" : 761658,\n'
                                             '        "iops" : 190414.083702,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1333089,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 38282,\n'
                                             '          "mean" : 939.539594,\n'
                                             '          "stddev" : '
                                             '143.340985,\n'
                                             '          "N" : 1333089\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1283,\n'
                                             '          "max" : 151294,\n'
                                             '          "mean" : '
                                             '19526.900070,\n'
                                             '          "stddev" : '
                                             '6576.455932,\n'
                                             '          "N" : 1333092,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 11328,\n'
                                             '            "5.000000" : 12480,\n'
                                             '            "10.000000" : '
                                             '13120,\n'
                                             '            "20.000000" : '
                                             '16064,\n'
                                             '            "30.000000" : '
                                             '16512,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '18048,\n'
                                             '            "60.000000" : '
                                             '19328,\n'
                                             '            "70.000000" : '
                                             '20352,\n'
                                             '            "80.000000" : '
                                             '22912,\n'
                                             '            "90.000000" : '
                                             '24448,\n'
                                             '            "95.000000" : '
                                             '32640,\n'
                                             '            "99.000000" : '
                                             '44288,\n'
                                             '            "99.500000" : '
                                             '47872,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '93696,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10961,\n'
                                             '          "max" : 152466,\n'
                                             '          "mean" : '
                                             '20466.439579,\n'
                                             '          "stddev" : '
                                             '6586.533800,\n'
                                             '          "N" : 1333092\n'
                                             '        },\n'
                                             '        "bw_min" : 760152,\n'
                                             '        "bw_max" : 764288,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '761844.357143,\n'
                                             '        "bw_dev" : 972.547456,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 190038,\n'
                                             '        "iops_max" : 191072,\n'
                                             '        "iops_mean" : '
                                             '190461.071429,\n'
                                             '        "iops_stddev" : '
                                             '243.120883,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 19,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 19.228874,\n'
                                             '        "4" : 80.757024,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 65.134736,\n'
                                             '        "50" : 34.590939,\n'
                                             '        "100" : 0.244395,\n'
                                             '        "250" : 0.028730,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87573: Tue Oct 25 23:40:46 '
                                             '2022\n'
                                             '  read: IOPS=190k, BW=744MiB/s '
                                             '(780MB/s)(5207MiB/7001msec)\n'
                                             '    slat (nsec): min=651, '
                                             'max=38282, avg=939.54, '
                                             'stdev=143.34\n'
                                             '    clat (nsec): min=1283, '
                                             'max=151294, avg=19526.90, '
                                             'stdev=6576.46\n'
                                             '     lat (usec): min=10, '
                                             'max=152, avg=20.47, stdev= 6.59\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   12],  '
                                             '5.00th=[   13], 10.00th=[   14], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   17], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '19], 60.00th=[   20],\n'
                                             '     | 70.00th=[   21], '
                                             '80.00th=[   23], 90.00th=[   '
                                             '25], 95.00th=[   33],\n'
                                             '     | 99.00th=[   45], '
                                             '99.50th=[   48], 99.90th=[   '
                                             '80], 99.95th=[   94],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=760152, '
                                             'max=764288, per=100.00%, '
                                             'avg=761844.36, stdev=972.55, '
                                             'samples=14\n'
                                             '   iops        : min=190038, '
                                             'max=191072, avg=190461.07, '
                                             'stdev=243.12, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.01%, 20=65.13%, '
                                             '50=34.59%\n'
                                             '  lat (usec)   : 100=0.24%, '
                                             '250=0.03%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=19, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=19.2%, 4=80.8%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1333089,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=744MiB/s (780MB/s), '
                                             '744MiB/s-744MiB/s '
                                             '(780MB/s-780MB/s), io=5207MiB '
                                             '(5460MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.output'),
                                   'state': {'begin': 1666726956.862852,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.782865047454834,
                                             'end': 1666726957.6457171,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_222.state')},
                       'cmd_223': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.output'),
                                   'state': {'begin': 1666726959.6549582,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7848594188690186,
                                             'end': 1666726960.4398177,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_223.state')},
                       'cmd_224': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.output'),
                                   'state': {'begin': 1666726960.4417746,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.47867488861084,
                                             'end': 1666726971.9204495,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_224.state')},
                       'cmd_225': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734061,\n'
                                             '  "timestamp_ms" : '
                                             '1666734061334,\n'
                                             '  "time" : "Tue Oct 25 23:41:01 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5464141824,\n'
                                             '        "io_kbytes" : 5336076,\n'
                                             '        "bw_bytes" : 780480191,\n'
                                             '        "bw" : 762187,\n'
                                             '        "iops" : 190546.493358,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1334016,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 661,\n'
                                             '          "max" : 14577,\n'
                                             '          "mean" : 940.799248,\n'
                                             '          "stddev" : 97.471830,\n'
                                             '          "N" : 1334016\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 9919,\n'
                                             '          "max" : 193162,\n'
                                             '          "mean" : '
                                             '19508.399007,\n'
                                             '          "stddev" : '
                                             '6583.473141,\n'
                                             '          "N" : 1334019,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 11328,\n'
                                             '            "5.000000" : 12480,\n'
                                             '            "10.000000" : '
                                             '13120,\n'
                                             '            "20.000000" : '
                                             '15936,\n'
                                             '            "30.000000" : '
                                             '16512,\n'
                                             '            "40.000000" : '
                                             '17024,\n'
                                             '            "50.000000" : '
                                             '18048,\n'
                                             '            "60.000000" : '
                                             '19328,\n'
                                             '            "70.000000" : '
                                             '20352,\n'
                                             '            "80.000000" : '
                                             '22912,\n'
                                             '            "90.000000" : '
                                             '24448,\n'
                                             '            "95.000000" : '
                                             '32640,\n'
                                             '            "99.000000" : '
                                             '44288,\n'
                                             '            "99.500000" : '
                                             '47872,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '92672,\n'
                                             '            "99.990000" : '
                                             '107008\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10850,\n'
                                             '          "max" : 193833,\n'
                                             '          "mean" : '
                                             '20449.198191,\n'
                                             '          "stddev" : '
                                             '6592.555887,\n'
                                             '          "N" : 1334019\n'
                                             '        },\n'
                                             '        "bw_min" : 760784,\n'
                                             '        "bw_max" : 763832,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '762373.571429,\n'
                                             '        "bw_dev" : 891.857932,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 190196,\n'
                                             '        "iops_max" : 190958,\n'
                                             '        "iops_mean" : '
                                             '190593.357143,\n'
                                             '        "iops_stddev" : '
                                             '222.929312,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 18,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 19.315660,\n'
                                             '        "4" : 80.683590,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 65.097870,\n'
                                             '        "50" : 34.639989,\n'
                                             '        "100" : 0.232681,\n'
                                             '        "250" : 0.029610,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=87623: Tue Oct 25 23:41:01 '
                                             '2022\n'
                                             '  read: IOPS=191k, BW=744MiB/s '
                                             '(780MB/s)(5211MiB/7001msec)\n'
                                             '    slat (nsec): min=661, '
                                             'max=14577, avg=940.80, '
                                             'stdev=97.47\n'
                                             '    clat (usec): min=9, max=193, '
                                             'avg=19.51, stdev= 6.58\n'
                                             '     lat (usec): min=10, '
                                             'max=193, avg=20.45, stdev= 6.59\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   12],  '
                                             '5.00th=[   13], 10.00th=[   14], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   17], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '19], 60.00th=[   20],\n'
                                             '     | 70.00th=[   21], '
                                             '80.00th=[   23], 90.00th=[   '
                                             '25], 95.00th=[   33],\n'
                                             '     | 99.00th=[   45], '
                                             '99.50th=[   48], 99.90th=[   '
                                             '80], 99.95th=[   93],\n'
                                             '     | 99.99th=[  108]\n'
                                             '   bw (  KiB/s): min=760784, '
                                             'max=763832, per=100.00%, '
                                             'avg=762373.57, stdev=891.86, '
                                             'samples=14\n'
                                             '   iops        : min=190196, '
                                             'max=190958, avg=190593.36, '
                                             'stdev=222.93, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=65.10%, 50=34.64%, 100=0.23%, '
                                             '250=0.03%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=18, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=19.3%, 4=80.7%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1334016,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=744MiB/s (780MB/s), '
                                             '744MiB/s-744MiB/s '
                                             '(780MB/s-780MB/s), io=5211MiB '
                                             '(5464MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.output'),
                                   'state': {'begin': 1666726971.9233382,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7887589931488037,
                                             'end': 1666726972.7120972,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_225.state')},
                       'cmd_226': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.output'),
                                   'state': {'begin': 1666726973.7527087,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.784700870513916,
                                             'end': 1666726974.5374095,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_226.state')},
                       'cmd_227': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.output'),
                                   'state': {'begin': 1666726974.539415,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.03250789642334,
                                             'end': 1666726985.5719228,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_227.state')},
                       'cmd_228': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734075,\n'
                                             '  "timestamp_ms" : '
                                             '1666734075058,\n'
                                             '  "time" : "Tue Oct 25 23:41:15 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4150661120,\n'
                                             '        "io_kbytes" : 4053380,\n'
                                             '        "bw_bytes" : 592866893,\n'
                                             '        "bw" : 578971,\n'
                                             '        "iops" : 144742.465362,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1013342,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2234,\n'
                                             '          "max" : 35627,\n'
                                             '          "mean" : 2427.005221,\n'
                                             '          "stddev" : '
                                             '421.685182,\n'
                                             '          "N" : 1013342\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 160,\n'
                                             '          "max" : 159249,\n'
                                             '          "mean" : '
                                             '24692.870506,\n'
                                             '          "stddev" : '
                                             '8160.319348,\n'
                                             '          "N" : 1013345,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14400,\n'
                                             '            "5.000000" : 14528,\n'
                                             '            "10.000000" : '
                                             '14912,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '20864,\n'
                                             '            "50.000000" : '
                                             '28800,\n'
                                             '            "60.000000" : '
                                             '29056,\n'
                                             '            "70.000000" : '
                                             '29312,\n'
                                             '            "80.000000" : '
                                             '29824,\n'
                                             '            "90.000000" : '
                                             '30080,\n'
                                             '            "95.000000" : '
                                             '34560,\n'
                                             '            "99.000000" : '
                                             '52992,\n'
                                             '            "99.500000" : '
                                             '55552,\n'
                                             '            "99.900000" : '
                                             '80384,\n'
                                             '            "99.950000" : '
                                             '96768,\n'
                                             '            "99.990000" : '
                                             '113152\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13666,\n'
                                             '          "max" : 161533,\n'
                                             '          "mean" : '
                                             '27119.875680,\n'
                                             '          "stddev" : '
                                             '8241.903875,\n'
                                             '          "N" : 1013345\n'
                                             '        },\n'
                                             '        "bw_min" : 577330,\n'
                                             '        "bw_max" : 581224,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '579190.428571,\n'
                                             '        "bw_dev" : 1221.195424,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 144332,\n'
                                             '        "iops_max" : 145306,\n'
                                             '        "iops_mean" : '
                                             '144797.571429,\n'
                                             '        "iops_stddev" : '
                                             '305.357474,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.071429,\n'
                                             '      "sys_cpu" : 60.985714,\n'
                                             '      "ctx" : 259323,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 35.179239,\n'
                                             '        "50" : 63.216762,\n'
                                             '        "100" : 1.556039,\n'
                                             '        "250" : 0.043223,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1452241,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 21892,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 21892,\n'
                                             '      "util" : 99.049128\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=87662: Tue '
                                             'Oct 25 23:41:15 2022\n'
                                             '  read: IOPS=145k, BW=565MiB/s '
                                             '(593MB/s)(3958MiB/7001msec)\n'
                                             '    slat (nsec): min=2234, '
                                             'max=35627, avg=2427.01, '
                                             'stdev=421.69\n'
                                             '    clat (nsec): min=160, '
                                             'max=159249, avg=24692.87, '
                                             'stdev=8160.32\n'
                                             '     lat (usec): min=13, '
                                             'max=161, avg=27.12, stdev= 8.24\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   21], 50.00th=[   '
                                             '29], 60.00th=[   30],\n'
                                             '     | 70.00th=[   30], '
                                             '80.00th=[   30], 90.00th=[   '
                                             '31], 95.00th=[   35],\n'
                                             '     | 99.00th=[   53], '
                                             '99.50th=[   56], 99.90th=[   '
                                             '81], 99.95th=[   97],\n'
                                             '     | 99.99th=[  114]\n'
                                             '   bw (  KiB/s): min=577330, '
                                             'max=581224, per=100.00%, '
                                             'avg=579190.43, stdev=1221.20, '
                                             'samples=14\n'
                                             '   iops        : min=144332, '
                                             'max=145306, avg=144797.57, '
                                             'stdev=305.36, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=35.18%, 50=63.22%, '
                                             '100=1.56%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=15.07%, '
                                             'sys=60.99%, ctx=259323, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1013342,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=565MiB/s (593MB/s), '
                                             '565MiB/s-565MiB/s '
                                             '(593MB/s-593MB/s), io=3958MiB '
                                             '(4151MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1452241/0, '
                                             'merge=0/0, ticks=21892/0, '
                                             'in_queue=21892, util=99.05%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.output'),
                                   'state': {'begin': 1666726985.574558,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7973747253417969,
                                             'end': 1666726986.3719327,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_228.state')},
                       'cmd_229': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.output'),
                                   'state': {'begin': 1666726987.367063,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7926807403564453,
                                             'end': 1666726988.1597438,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_229.state')},
                       'cmd_23': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.output'),
                                  'state': {'begin': 1666726000.1902053,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.493699789047241,
                                            'end': 1666726011.6839051,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_23.state')},
                       'cmd_230': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.output'),
                                   'state': {'begin': 1666726988.1618998,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.053723812103271,
                                             'end': 1666726999.2156236,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_230.state')},
                       'cmd_231': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734088,\n'
                                             '  "timestamp_ms" : '
                                             '1666734088701,\n'
                                             '  "time" : "Tue Oct 25 23:41:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4203417600,\n'
                                             '        "io_kbytes" : 4104900,\n'
                                             '        "bw_bytes" : 600402456,\n'
                                             '        "bw" : 586330,\n'
                                             '        "iops" : 146582.202542,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1026222,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2174,\n'
                                             '          "max" : 178164,\n'
                                             '          "mean" : 2386.173347,\n'
                                             '          "stddev" : '
                                             '455.786112,\n'
                                             '          "N" : 1026222\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 191,\n'
                                             '          "max" : 208732,\n'
                                             '          "mean" : '
                                             '24370.688623,\n'
                                             '          "stddev" : '
                                             '8077.609207,\n'
                                             '          "N" : 1026225,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14272,\n'
                                             '            "5.000000" : 15040,\n'
                                             '            "10.000000" : '
                                             '16512,\n'
                                             '            "20.000000" : '
                                             '16768,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17792,\n'
                                             '            "50.000000" : '
                                             '28544,\n'
                                             '            "60.000000" : '
                                             '28800,\n'
                                             '            "70.000000" : '
                                             '28800,\n'
                                             '            "80.000000" : '
                                             '29056,\n'
                                             '            "90.000000" : '
                                             '29568,\n'
                                             '            "95.000000" : '
                                             '34560,\n'
                                             '            "99.000000" : '
                                             '52480,\n'
                                             '            "99.500000" : '
                                             '54016,\n'
                                             '            "99.900000" : '
                                             '83456,\n'
                                             '            "99.950000" : '
                                             '99840,\n'
                                             '            "99.990000" : '
                                             '115200\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11301,\n'
                                             '          "max" : 217257,\n'
                                             '          "mean" : '
                                             '26756.861984,\n'
                                             '          "stddev" : '
                                             '8171.492717,\n'
                                             '          "N" : 1026225\n'
                                             '        },\n'
                                             '        "bw_min" : 584704,\n'
                                             '        "bw_max" : 588312,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '586556.714286,\n'
                                             '        "bw_dev" : 891.194395,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 146176,\n'
                                             '        "iops_max" : 147078,\n'
                                             '        "iops_mean" : '
                                             '146639.000000,\n'
                                             '        "iops_stddev" : '
                                             '222.618681,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.600000,\n'
                                             '      "sys_cpu" : 60.242857,\n'
                                             '      "ctx" : 262841,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 41.038976,\n'
                                             '        "50" : 57.387485,\n'
                                             '        "100" : 1.520529,\n'
                                             '        "250" : 0.049210,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1469749,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 22115,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 22115,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=87701: Tue '
                                             'Oct 25 23:41:28 2022\n'
                                             '  read: IOPS=147k, BW=573MiB/s '
                                             '(600MB/s)(4009MiB/7001msec)\n'
                                             '    slat (nsec): min=2174, '
                                             'max=178164, avg=2386.17, '
                                             'stdev=455.79\n'
                                             '    clat (nsec): min=191, '
                                             'max=208732, avg=24370.69, '
                                             'stdev=8077.61\n'
                                             '     lat (usec): min=11, '
                                             'max=217, avg=26.76, stdev= 8.17\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   16], 10.00th=[   17], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '29], 60.00th=[   29],\n'
                                             '     | 70.00th=[   29], '
                                             '80.00th=[   30], 90.00th=[   '
                                             '30], 95.00th=[   35],\n'
                                             '     | 99.00th=[   53], '
                                             '99.50th=[   55], 99.90th=[   '
                                             '84], 99.95th=[  100],\n'
                                             '     | 99.99th=[  116]\n'
                                             '   bw (  KiB/s): min=584704, '
                                             'max=588312, per=100.00%, '
                                             'avg=586556.71, stdev=891.19, '
                                             'samples=14\n'
                                             '   iops        : min=146176, '
                                             'max=147078, avg=146639.00, '
                                             'stdev=222.62, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=41.04%, 50=57.39%, '
                                             '100=1.52%\n'
                                             '  lat (usec)   : 250=0.05%\n'
                                             '  cpu          : usr=15.60%, '
                                             'sys=60.24%, ctx=262841, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1026222,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=573MiB/s (600MB/s), '
                                             '573MiB/s-573MiB/s '
                                             '(600MB/s-600MB/s), io=4009MiB '
                                             '(4203MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1469749/0, '
                                             'merge=0/0, ticks=22115/0, '
                                             'in_queue=22115, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.output'),
                                   'state': {'begin': 1666726999.218482,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7940006256103516,
                                             'end': 1666727000.0124826,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_231.state')},
                       'cmd_232': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.output'),
                                   'state': {'begin': 1666727001.0083597,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7922351360321045,
                                             'end': 1666727001.8005948,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_232.state')},
                       'cmd_233': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.output'),
                                   'state': {'begin': 1666727001.8026242,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.044522523880005,
                                             'end': 1666727012.8471467,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_233.state')},
                       'cmd_234': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734102,\n'
                                             '  "timestamp_ms" : '
                                             '1666734102332,\n'
                                             '  "time" : "Tue Oct 25 23:41:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4528324608,\n'
                                             '        "io_kbytes" : 4422192,\n'
                                             '        "bw_bytes" : 646811113,\n'
                                             '        "bw" : 631651,\n'
                                             '        "iops" : 157912.441080,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1105545,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2275,\n'
                                             '          "max" : 119033,\n'
                                             '          "mean" : 2745.671780,\n'
                                             '          "stddev" : '
                                             '595.147245,\n'
                                             '          "N" : 1105545\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 3747,\n'
                                             '          "max" : 138570,\n'
                                             '          "mean" : '
                                             '22051.316446,\n'
                                             '          "stddev" : '
                                             '5281.925427,\n'
                                             '          "N" : 1105548,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 17792,\n'
                                             '            "5.000000" : 19328,\n'
                                             '            "10.000000" : '
                                             '19328,\n'
                                             '            "20.000000" : '
                                             '19840,\n'
                                             '            "30.000000" : '
                                             '20864,\n'
                                             '            "40.000000" : '
                                             '20864,\n'
                                             '            "50.000000" : '
                                             '21120,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '22400,\n'
                                             '            "95.000000" : '
                                             '33024,\n'
                                             '            "99.000000" : '
                                             '47360,\n'
                                             '            "99.500000" : '
                                             '50432,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '93696,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12774,\n'
                                             '          "max" : 140935,\n'
                                             '          "mean" : '
                                             '24796.987255,\n'
                                             '          "stddev" : '
                                             '5227.303506,\n'
                                             '          "N" : 1105548\n'
                                             '        },\n'
                                             '        "bw_min" : 630016,\n'
                                             '        "bw_max" : 633688,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '632033.142857,\n'
                                             '        "bw_dev" : 1130.843651,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 157504,\n'
                                             '        "iops_max" : 158422,\n'
                                             '        "iops_mean" : '
                                             '158008.214286,\n'
                                             '        "iops_stddev" : '
                                             '282.708923,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.085714,\n'
                                             '      "sys_cpu" : 57.100000,\n'
                                             '      "ctx" : 284692,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 1,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 21.244273,\n'
                                             '        "50" : 78.183339,\n'
                                             '        "100" : 0.536839,\n'
                                             '        "250" : 0.034915,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1564440,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 23344,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 23344,\n'
                                             '      "util" : 99.108381\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=87740: Tue '
                                             'Oct 25 23:41:42 2022\n'
                                             '  read: IOPS=158k, BW=617MiB/s '
                                             '(647MB/s)(4319MiB/7001msec)\n'
                                             '    slat (nsec): min=2275, '
                                             'max=119033, avg=2745.67, '
                                             'stdev=595.15\n'
                                             '    clat (usec): min=3, max=138, '
                                             'avg=22.05, stdev= 5.28\n'
                                             '     lat (usec): min=12, '
                                             'max=140, avg=24.80, stdev= 5.23\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   18],  '
                                             '5.00th=[   20], 10.00th=[   20], '
                                             '20.00th=[   20],\n'
                                             '     | 30.00th=[   21], '
                                             '40.00th=[   21], 50.00th=[   '
                                             '22], 60.00th=[   22],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '23], 95.00th=[   34],\n'
                                             '     | 99.00th=[   48], '
                                             '99.50th=[   51], 99.90th=[   '
                                             '80], 99.95th=[   94],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=630016, '
                                             'max=633688, per=100.00%, '
                                             'avg=632033.14, stdev=1130.84, '
                                             'samples=14\n'
                                             '   iops        : min=157504, '
                                             'max=158422, avg=158008.21, '
                                             'stdev=282.71, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=21.24%, 50=78.18%, '
                                             '100=0.54%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=14.09%, '
                                             'sys=57.10%, ctx=284692, majf=0, '
                                             'minf=1\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1105545,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=617MiB/s (647MB/s), '
                                             '617MiB/s-617MiB/s '
                                             '(647MB/s-647MB/s), io=4319MiB '
                                             '(4528MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1564440/0, '
                                             'merge=0/0, ticks=23344/0, '
                                             'in_queue=23344, util=99.11%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.output'),
                                   'state': {'begin': 1666727012.849836,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7898743152618408,
                                             'end': 1666727013.6397104,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_234.state')},
                       'cmd_235': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.output'),
                                   'state': {'begin': 1666727015.6616108,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7738947868347168,
                                             'end': 1666727016.4355056,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_235.state')},
                       'cmd_236': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.output'),
                                   'state': {'begin': 1666727016.437498,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.448853969573975,
                                             'end': 1666727027.886352,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_236.state')},
                       'cmd_237': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734117,\n'
                                             '  "timestamp_ms" : '
                                             '1666734117300,\n'
                                             '  "time" : "Tue Oct 25 23:41:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5844619264,\n'
                                             '        "io_kbytes" : 5707636,\n'
                                             '        "bw_bytes" : 834826348,\n'
                                             '        "bw" : 815260,\n'
                                             '        "iops" : 203814.597915,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1426906,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 15208,\n'
                                             '          "mean" : 166.282201,\n'
                                             '          "stddev" : 53.441501,\n'
                                             '          "N" : 1426906\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 10440,\n'
                                             '          "max" : 145372,\n'
                                             '          "mean" : '
                                             '19068.696730,\n'
                                             '          "stddev" : '
                                             '5673.395713,\n'
                                             '          "N" : 1426908,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 15424,\n'
                                             '            "5.000000" : 16320,\n'
                                             '            "10.000000" : '
                                             '17024,\n'
                                             '            "20.000000" : '
                                             '17536,\n'
                                             '            "30.000000" : '
                                             '17536,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '17792,\n'
                                             '            "60.000000" : '
                                             '17792,\n'
                                             '            "70.000000" : '
                                             '18048,\n'
                                             '            "80.000000" : '
                                             '18048,\n'
                                             '            "90.000000" : '
                                             '19584,\n'
                                             '            "95.000000" : '
                                             '31104,\n'
                                             '            "99.000000" : '
                                             '43776,\n'
                                             '            "99.500000" : '
                                             '46336,\n'
                                             '            "99.900000" : '
                                             '83456,\n'
                                             '            "99.950000" : '
                                             '98816,\n'
                                             '            "99.990000" : '
                                             '123392\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10620,\n'
                                             '          "max" : 145553,\n'
                                             '          "mean" : '
                                             '19234.978923,\n'
                                             '          "stddev" : '
                                             '5673.110075,\n'
                                             '          "N" : 1426908\n'
                                             '        },\n'
                                             '        "bw_min" : 812432,\n'
                                             '        "bw_max" : 821851,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '815661.571429,\n'
                                             '        "bw_dev" : 2674.021567,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 203108,\n'
                                             '        "iops_max" : 205462,\n'
                                             '        "iops_mean" : '
                                             '203915.428571,\n'
                                             '        "iops_stddev" : '
                                             '668.389306,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 60.214286,\n'
                                             '      "sys_cpu" : 39.757143,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 30.973379,\n'
                                             '        "4" : 69.014077,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 91.161156,\n'
                                             '        "50" : 8.538124,\n'
                                             '        "100" : 0.253556,\n'
                                             '        "250" : 0.047305,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=87788: Tue '
                                             'Oct 25 23:41:57 2022\n'
                                             '  read: IOPS=204k, BW=796MiB/s '
                                             '(835MB/s)(5574MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=15208, avg=166.28, '
                                             'stdev=53.44\n'
                                             '    clat (usec): min=10, '
                                             'max=145, avg=19.07, stdev= 5.67\n'
                                             '     lat (usec): min=10, '
                                             'max=145, avg=19.23, stdev= 5.67\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   16],  '
                                             '5.00th=[   17], 10.00th=[   18], '
                                             '20.00th=[   18],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '18], 60.00th=[   18],\n'
                                             '     | 70.00th=[   19], '
                                             '80.00th=[   19], 90.00th=[   '
                                             '20], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   47], 99.90th=[   '
                                             '84], 99.95th=[   99],\n'
                                             '     | 99.99th=[  124]\n'
                                             '   bw (  KiB/s): min=812432, '
                                             'max=821851, per=100.00%, '
                                             'avg=815661.57, stdev=2674.02, '
                                             'samples=14\n'
                                             '   iops        : min=203108, '
                                             'max=205462, avg=203915.43, '
                                             'stdev=668.39, samples=14\n'
                                             '  lat (usec)   : 20=91.16%, '
                                             '50=8.54%, 100=0.25%, 250=0.05%\n'
                                             '  cpu          : usr=60.21%, '
                                             'sys=39.76%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, '
                                             '2=31.0%, 4=69.0%, 8=0.0%, '
                                             '16=0.0%, 32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1426906,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=796MiB/s (835MB/s), '
                                             '796MiB/s-796MiB/s '
                                             '(835MB/s-835MB/s), io=5574MiB '
                                             '(5845MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.output'),
                                   'state': {'begin': 1666727027.887692,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7917938232421875,
                                             'end': 1666727028.6794858,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_237.state')},
                       'cmd_238': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.output'),
                                   'state': {'begin': 1666727030.7304463,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7934126853942871,
                                             'end': 1666727031.523859,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_238.state')},
                       'cmd_239': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.output'),
                                   'state': {'begin': 1666727031.5263135,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.473499536514282,
                                             'end': 1666727042.999813,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_239.state')},
                       'cmd_24': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733101,\n'
                                            '  "timestamp_ms" : '
                                            '1666733101115,\n'
                                            '  "time" : "Tue Oct 25 23:25:01 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2411470848,\n'
                                            '        "io_kbytes" : 2354952,\n'
                                            '        "bw_bytes" : 344446628,\n'
                                            '        "bw" : 336373,\n'
                                            '        "iops" : 84093.415226,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 588738,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 711,\n'
                                            '          "max" : 25067,\n'
                                            '          "mean" : 928.008393,\n'
                                            '          "stddev" : 146.987262,\n'
                                            '          "N" : 588738\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 701,\n'
                                            '          "max" : 113202,\n'
                                            '          "mean" : 10713.652358,\n'
                                            '          "stddev" : '
                                            '2206.439842,\n'
                                            '          "N" : 588738,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9920,\n'
                                            '            "5.000000" : 10048,\n'
                                            '            "10.000000" : 10176,\n'
                                            '            "20.000000" : 10176,\n'
                                            '            "30.000000" : 10304,\n'
                                            '            "40.000000" : 10304,\n'
                                            '            "50.000000" : 10432,\n'
                                            '            "60.000000" : 10432,\n'
                                            '            "70.000000" : 10432,\n'
                                            '            "80.000000" : 10560,\n'
                                            '            "90.000000" : 10688,\n'
                                            '            "95.000000" : 11200,\n'
                                            '            "99.000000" : 20096,\n'
                                            '            "99.500000" : 27520,\n'
                                            '            "99.900000" : 30592,\n'
                                            '            "99.950000" : 43776,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 10590,\n'
                                            '          "max" : 114104,\n'
                                            '          "mean" : 11641.660751,\n'
                                            '          "stddev" : '
                                            '2210.286923,\n'
                                            '          "N" : 588738\n'
                                            '        },\n'
                                            '        "bw_min" : 335992,\n'
                                            '        "bw_max" : 336976,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '336465.214286,\n'
                                            '        "bw_dev" : 296.234361,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83998,\n'
                                            '        "iops_max" : 84244,\n'
                                            '        "iops_mean" : '
                                            '84116.285714,\n'
                                            '        "iops_stddev" : '
                                            '74.034741,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 15,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 1.542282,\n'
                                            '        "20" : 97.451498,\n'
                                            '        "50" : 0.959000,\n'
                                            '        "100" : 0.044842,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84683: Tue Oct 25 23:25:01 '
                                            '2022\n'
                                            '  read: IOPS=84.1k, BW=328MiB/s '
                                            '(344MB/s)(2300MiB/7001msec)\n'
                                            '    slat (nsec): min=711, '
                                            'max=25067, avg=928.01, '
                                            'stdev=146.99\n'
                                            '    clat (nsec): min=701, '
                                            'max=113202, avg=10713.65, '
                                            'stdev=2206.44\n'
                                            '     lat (usec): min=10, max=114, '
                                            'avg=11.64, stdev= 2.21\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9920],  '
                                            '5.00th=[10048], 10.00th=[10176], '
                                            '20.00th=[10176],\n'
                                            '     | 30.00th=[10304], '
                                            '40.00th=[10304], 50.00th=[10432], '
                                            '60.00th=[10432],\n'
                                            '     | 70.00th=[10432], '
                                            '80.00th=[10560], 90.00th=[10688], '
                                            '95.00th=[11200],\n'
                                            '     | 99.00th=[20096], '
                                            '99.50th=[27520], 99.90th=[30592], '
                                            '99.95th=[43776],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=335992, '
                                            'max=336976, per=100.00%, '
                                            'avg=336465.21, stdev=296.23, '
                                            'samples=14\n'
                                            '   iops        : min=83998, '
                                            'max=84244, avg=84116.29, '
                                            'stdev=74.03, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=1.54%, 20=97.45%, '
                                            '50=0.96%\n'
                                            '  lat (usec)   : 100=0.04%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=15, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=588738,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=328MiB/s (344MB/s), '
                                            '328MiB/s-328MiB/s '
                                            '(344MB/s-344MB/s), io=2300MiB '
                                            '(2411MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.output'),
                                  'state': {'begin': 1666726011.6866117,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7878735065460205,
                                            'end': 1666726012.4744852,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_24.state')},
                       'cmd_240': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734132,\n'
                                             '  "timestamp_ms" : '
                                             '1666734132411,\n'
                                             '  "time" : "Tue Oct 25 23:42:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6327066624,\n'
                                             '        "io_kbytes" : 6178776,\n'
                                             '        "bw_bytes" : 903737555,\n'
                                             '        "bw" : 882556,\n'
                                             '        "iops" : 220638.623054,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1544691,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17432,\n'
                                             '          "mean" : 189.121560,\n'
                                             '          "stddev" : '
                                             '121.269366,\n'
                                             '          "N" : 1544691\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 9367,\n'
                                             '          "max" : 202460,\n'
                                             '          "mean" : '
                                             '17679.470528,\n'
                                             '          "stddev" : '
                                             '5728.764872,\n'
                                             '          "N" : 1544694,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 11072,\n'
                                             '            "5.000000" : 15296,\n'
                                             '            "10.000000" : '
                                             '15936,\n'
                                             '            "20.000000" : '
                                             '16064,\n'
                                             '            "30.000000" : '
                                             '16064,\n'
                                             '            "40.000000" : '
                                             '16064,\n'
                                             '            "50.000000" : '
                                             '16192,\n'
                                             '            "60.000000" : '
                                             '16192,\n'
                                             '            "70.000000" : '
                                             '16320,\n'
                                             '            "80.000000" : '
                                             '16512,\n'
                                             '            "90.000000" : '
                                             '20608,\n'
                                             '            "95.000000" : '
                                             '29312,\n'
                                             '            "99.000000" : '
                                             '39168,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '92672,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9678,\n'
                                             '          "max" : 202640,\n'
                                             '          "mean" : '
                                             '17868.592084,\n'
                                             '          "stddev" : '
                                             '5727.259130,\n'
                                             '          "N" : 1544694\n'
                                             '        },\n'
                                             '        "bw_min" : 880480,\n'
                                             '        "bw_max" : 884112,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '882771.142857,\n'
                                             '        "bw_dev" : 946.312914,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 220120,\n'
                                             '        "iops_max" : 221028,\n'
                                             '        "iops_mean" : '
                                             '220692.928571,\n'
                                             '        "iops_stddev" : '
                                             '236.341495,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 42.900000,\n'
                                             '      "sys_cpu" : 57.071429,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.363309,\n'
                                             '        "2" : 2.448580,\n'
                                             '        "4" : 97.188111,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 89.582124,\n'
                                             '        "50" : 10.162874,\n'
                                             '        "100" : 0.218814,\n'
                                             '        "250" : 0.032369,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=87838: Tue '
                                             'Oct 25 23:42:12 2022\n'
                                             '  read: IOPS=221k, BW=862MiB/s '
                                             '(904MB/s)(6034MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17432, avg=189.12, '
                                             'stdev=121.27\n'
                                             '    clat (usec): min=9, max=202, '
                                             'avg=17.68, stdev= 5.73\n'
                                             '     lat (usec): min=9, max=202, '
                                             'avg=17.87, stdev= 5.73\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   12],  '
                                             '5.00th=[   16], 10.00th=[   16], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   17], '
                                             '40.00th=[   17], 50.00th=[   '
                                             '17], 60.00th=[   17],\n'
                                             '     | 70.00th=[   17], '
                                             '80.00th=[   17], 90.00th=[   '
                                             '21], 95.00th=[   30],\n'
                                             '     | 99.00th=[   40], '
                                             '99.50th=[   45], 99.90th=[   '
                                             '82], 99.95th=[   93],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=880480, '
                                             'max=884112, per=100.00%, '
                                             'avg=882771.14, stdev=946.31, '
                                             'samples=14\n'
                                             '   iops        : min=220120, '
                                             'max=221028, avg=220692.93, '
                                             'stdev=236.34, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=89.58%, 50=10.16%, 100=0.22%, '
                                             '250=0.03%\n'
                                             '  cpu          : usr=42.90%, '
                                             'sys=57.07%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, 2=2.4%, '
                                             '4=97.2%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1544691,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=862MiB/s (904MB/s), '
                                             '862MiB/s-862MiB/s '
                                             '(904MB/s-904MB/s), io=6034MiB '
                                             '(6327MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.output'),
                                   'state': {'begin': 1666727043.0025334,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7948098182678223,
                                             'end': 1666727043.7973433,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_240.state')},
                       'cmd_241': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.output'),
                                   'state': {'begin': 1666727045.8665073,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7829034328460693,
                                             'end': 1666727046.6494107,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_241.state')},
                       'cmd_242': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.output'),
                                   'state': {'begin': 1666727046.6566803,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.461086511611938,
                                             'end': 1666727058.1177669,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_242.state')},
                       'cmd_243': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734147,\n'
                                             '  "timestamp_ms" : '
                                             '1666734147528,\n'
                                             '  "time" : "Tue Oct 25 23:42:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6299238400,\n'
                                             '        "io_kbytes" : 6151600,\n'
                                             '        "bw_bytes" : 899762662,\n'
                                             '        "bw" : 878674,\n'
                                             '        "iops" : 219668.190259,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1537897,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 16481,\n'
                                             '          "mean" : 214.052651,\n'
                                             '          "stddev" : '
                                             '212.685902,\n'
                                             '          "N" : 1537897\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 9428,\n'
                                             '          "max" : 196909,\n'
                                             '          "mean" : '
                                             '17698.782211,\n'
                                             '          "stddev" : '
                                             '5860.452451,\n'
                                             '          "N" : 1537900,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 11072,\n'
                                             '            "5.000000" : 14144,\n'
                                             '            "10.000000" : '
                                             '15424,\n'
                                             '            "20.000000" : '
                                             '15936,\n'
                                             '            "30.000000" : '
                                             '16064,\n'
                                             '            "40.000000" : '
                                             '16064,\n'
                                             '            "50.000000" : '
                                             '16192,\n'
                                             '            "60.000000" : '
                                             '16192,\n'
                                             '            "70.000000" : '
                                             '16512,\n'
                                             '            "80.000000" : '
                                             '17280,\n'
                                             '            "90.000000" : '
                                             '21120,\n'
                                             '            "95.000000" : '
                                             '29568,\n'
                                             '            "99.000000" : '
                                             '41216,\n'
                                             '            "99.500000" : '
                                             '44288,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '93696,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9678,\n'
                                             '          "max" : 197099,\n'
                                             '          "mean" : '
                                             '17912.834803,\n'
                                             '          "stddev" : '
                                             '5855.839323,\n'
                                             '          "N" : 1537900\n'
                                             '        },\n'
                                             '        "bw_min" : 875880,\n'
                                             '        "bw_max" : 881400,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '878894.857143,\n'
                                             '        "bw_dev" : 1748.287903,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 218970,\n'
                                             '        "iops_max" : 220350,\n'
                                             '        "iops_mean" : '
                                             '219723.714286,\n'
                                             '        "iops_stddev" : '
                                             '437.071976,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 43.414286,\n'
                                             '      "sys_cpu" : 56.557143,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.328696,\n'
                                             '        "2" : 5.216539,\n'
                                             '        "4" : 94.454765,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 87.246285,\n'
                                             '        "50" : 12.507144,\n'
                                             '        "100" : 0.209442,\n'
                                             '        "250" : 0.031602,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=87889: Tue '
                                             'Oct 25 23:42:27 2022\n'
                                             '  read: IOPS=220k, BW=858MiB/s '
                                             '(900MB/s)(6007MiB/7001msec)\n'
                                             '    slat (nsec): min=120, '
                                             'max=16481, avg=214.05, '
                                             'stdev=212.69\n'
                                             '    clat (usec): min=9, max=196, '
                                             'avg=17.70, stdev= 5.86\n'
                                             '     lat (usec): min=9, max=197, '
                                             'avg=17.91, stdev= 5.86\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   12],  '
                                             '5.00th=[   15], 10.00th=[   16], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   17], '
                                             '40.00th=[   17], 50.00th=[   '
                                             '17], 60.00th=[   17],\n'
                                             '     | 70.00th=[   17], '
                                             '80.00th=[   18], 90.00th=[   '
                                             '22], 95.00th=[   30],\n'
                                             '     | 99.00th=[   42], '
                                             '99.50th=[   45], 99.90th=[   '
                                             '82], 99.95th=[   94],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=875880, '
                                             'max=881400, per=100.00%, '
                                             'avg=878894.86, stdev=1748.29, '
                                             'samples=14\n'
                                             '   iops        : min=218970, '
                                             'max=220350, avg=219723.71, '
                                             'stdev=437.07, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=87.25%, 50=12.51%, 100=0.21%, '
                                             '250=0.03%\n'
                                             '  cpu          : usr=43.41%, '
                                             'sys=56.56%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.3%, 2=5.2%, '
                                             '4=94.5%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1537897,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=858MiB/s (900MB/s), '
                                             '858MiB/s-858MiB/s '
                                             '(900MB/s-900MB/s), io=6007MiB '
                                             '(6299MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.output'),
                                   'state': {'begin': 1666727058.1210501,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.792534589767456,
                                             'end': 1666727058.9135847,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_243.state')},
                       'cmd_244': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.output'),
                                   'state': {'begin': 1666727060.973781,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7824304103851318,
                                             'end': 1666727061.7562115,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_244.state')},
                       'cmd_245': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.output'),
                                   'state': {'begin': 1666727061.7584105,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.355727434158325,
                                             'end': 1666727073.114138,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_245.state')},
                       'cmd_246': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734162,\n'
                                             '  "timestamp_ms" : '
                                             '1666734162525,\n'
                                             '  "time" : "Tue Oct 25 23:42:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6373732352,\n'
                                             '        "io_kbytes" : 6224348,\n'
                                             '        "bw_bytes" : 910403135,\n'
                                             '        "bw" : 889065,\n'
                                             '        "iops" : 222265.962005,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1556084,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1462,\n'
                                             '          "max" : 37179,\n'
                                             '          "mean" : 1641.605073,\n'
                                             '          "stddev" : '
                                             '199.551002,\n'
                                             '          "N" : 1556084\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 126758,\n'
                                             '          "mean" : '
                                             '16115.646010,\n'
                                             '          "stddev" : '
                                             '5418.188609,\n'
                                             '          "N" : 1556087,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14144,\n'
                                             '            "5.000000" : 14400,\n'
                                             '            "10.000000" : '
                                             '14528,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '14656,\n'
                                             '            "40.000000" : '
                                             '14656,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '14784,\n'
                                             '            "80.000000" : '
                                             '14912,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '36096,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '78336,\n'
                                             '            "99.950000" : '
                                             '88576,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10790,\n'
                                             '          "max" : 128381,\n'
                                             '          "mean" : '
                                             '17757.251002,\n'
                                             '          "stddev" : '
                                             '5418.749748,\n'
                                             '          "N" : 1556087\n'
                                             '        },\n'
                                             '        "bw_min" : 888000,\n'
                                             '        "bw_max" : 890464,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '889269.357143,\n'
                                             '        "bw_dev" : 942.858143,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 222000,\n'
                                             '        "iops_max" : 222616,\n'
                                             '        "iops_mean" : '
                                             '222317.285714,\n'
                                             '        "iops_stddev" : '
                                             '235.776435,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.328571,\n'
                                             '      "sys_cpu" : 81.642857,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 91.798450,\n'
                                             '        "50" : 8.025659,\n'
                                             '        "100" : 0.146200,\n'
                                             '        "250" : 0.019986,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87938: Tue '
                                             'Oct 25 23:42:42 2022\n'
                                             '  read: IOPS=222k, BW=868MiB/s '
                                             '(910MB/s)(6078MiB/7001msec)\n'
                                             '    slat (nsec): min=1462, '
                                             'max=37179, avg=1641.61, '
                                             'stdev=199.55\n'
                                             '    clat (nsec): min=230, '
                                             'max=126758, avg=16115.65, '
                                             'stdev=5418.19\n'
                                             '     lat (usec): min=10, '
                                             'max=128, avg=17.76, stdev= 5.42\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   15], '
                                             '80.00th=[   15], 90.00th=[   '
                                             '17], 95.00th=[   28],\n'
                                             '     | 99.00th=[   37], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '79], 99.95th=[   89],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=888000, '
                                             'max=890464, per=100.00%, '
                                             'avg=889269.36, stdev=942.86, '
                                             'samples=14\n'
                                             '   iops        : min=222000, '
                                             'max=222616, avg=222317.29, '
                                             'stdev=235.78, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=91.80%, 50=8.03%, '
                                             '100=0.15%\n'
                                             '  lat (usec)   : 250=0.02%\n'
                                             '  cpu          : usr=18.33%, '
                                             'sys=81.64%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1556084,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=868MiB/s (910MB/s), '
                                             '868MiB/s-868MiB/s '
                                             '(910MB/s-910MB/s), io=6078MiB '
                                             '(6374MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.output'),
                                   'state': {'begin': 1666727073.1173422,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7967362403869629,
                                             'end': 1666727073.9140785,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_246.state')},
                       'cmd_247': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.output'),
                                   'state': {'begin': 1666727075.930548,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7862098217010498,
                                             'end': 1666727076.7167578,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_247.state')},
                       'cmd_248': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.output'),
                                   'state': {'begin': 1666727076.7187743,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.360539197921753,
                                             'end': 1666727088.0793135,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_248.state')},
                       'cmd_249': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734177,\n'
                                             '  "timestamp_ms" : '
                                             '1666734177491,\n'
                                             '  "time" : "Tue Oct 25 23:42:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6295175168,\n'
                                             '        "io_kbytes" : 6147632,\n'
                                             '        "bw_bytes" : 899182283,\n'
                                             '        "bw" : 878107,\n'
                                             '        "iops" : 219526.496215,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1536905,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1593,\n'
                                             '          "max" : 27100,\n'
                                             '          "mean" : 1862.124565,\n'
                                             '          "stddev" : '
                                             '155.390262,\n'
                                             '          "N" : 1536905\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 250,\n'
                                             '          "max" : 140985,\n'
                                             '          "mean" : '
                                             '16110.551588,\n'
                                             '          "stddev" : '
                                             '6695.143158,\n'
                                             '          "N" : 1536908,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10432,\n'
                                             '            "5.000000" : 10432,\n'
                                             '            "10.000000" : '
                                             '10560,\n'
                                             '            "20.000000" : '
                                             '12480,\n'
                                             '            "30.000000" : '
                                             '12736,\n'
                                             '            "40.000000" : '
                                             '13248,\n'
                                             '            "50.000000" : '
                                             '14400,\n'
                                             '            "60.000000" : '
                                             '14528,\n'
                                             '            "70.000000" : '
                                             '16768,\n'
                                             '            "80.000000" : '
                                             '20608,\n'
                                             '            "90.000000" : '
                                             '21376,\n'
                                             '            "95.000000" : '
                                             '29312,\n'
                                             '            "99.000000" : '
                                             '41216,\n'
                                             '            "99.500000" : '
                                             '44800,\n'
                                             '            "99.900000" : '
                                             '80384,\n'
                                             '            "99.950000" : '
                                             '96768,\n'
                                             '            "99.990000" : '
                                             '122368\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9629,\n'
                                             '          "max" : 142838,\n'
                                             '          "mean" : '
                                             '17972.676267,\n'
                                             '          "stddev" : '
                                             '6706.536732,\n'
                                             '          "N" : 1536908\n'
                                             '        },\n'
                                             '        "bw_min" : 876896,\n'
                                             '        "bw_max" : 880945,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '878672.428571,\n'
                                             '        "bw_dev" : 1464.370472,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 219224,\n'
                                             '        "iops_max" : 220236,\n'
                                             '        "iops_mean" : '
                                             '219668.071429,\n'
                                             '        "iops_stddev" : '
                                             '366.064405,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.700000,\n'
                                             '      "sys_cpu" : 82.271429,\n'
                                             '      "ctx" : 18,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.043334,\n'
                                             '        "20" : 76.322609,\n'
                                             '        "50" : 23.389604,\n'
                                             '        "100" : 0.200533,\n'
                                             '        "250" : 0.043919,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=87987: Tue '
                                             'Oct 25 23:42:57 2022\n'
                                             '  read: IOPS=220k, BW=858MiB/s '
                                             '(899MB/s)(6004MiB/7001msec)\n'
                                             '    slat (nsec): min=1593, '
                                             'max=27100, avg=1862.12, '
                                             'stdev=155.39\n'
                                             '    clat (nsec): min=250, '
                                             'max=140985, avg=16110.55, '
                                             'stdev=6695.14\n'
                                             '     lat (usec): min=9, max=142, '
                                             'avg=17.97, stdev= 6.71\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   11],  '
                                             '5.00th=[   11], 10.00th=[   11], '
                                             '20.00th=[   13],\n'
                                             '     | 30.00th=[   13], '
                                             '40.00th=[   14], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   17], '
                                             '80.00th=[   21], 90.00th=[   '
                                             '22], 95.00th=[   30],\n'
                                             '     | 99.00th=[   42], '
                                             '99.50th=[   45], 99.90th=[   '
                                             '81], 99.95th=[   97],\n'
                                             '     | 99.99th=[  123]\n'
                                             '   bw (  KiB/s): min=876896, '
                                             'max=880945, per=100.00%, '
                                             'avg=878672.43, stdev=1464.37, '
                                             'samples=14\n'
                                             '   iops        : min=219224, '
                                             'max=220236, avg=219668.07, '
                                             'stdev=366.06, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=0.04%, '
                                             '20=76.32%, 50=23.39%, 100=0.20%, '
                                             '250=0.04%\n'
                                             '  cpu          : usr=17.70%, '
                                             'sys=82.27%, ctx=18, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1536905,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=858MiB/s (899MB/s), '
                                             '858MiB/s-858MiB/s '
                                             '(899MB/s-899MB/s), io=6004MiB '
                                             '(6295MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.output'),
                                   'state': {'begin': 1666727088.0860672,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7894282341003418,
                                             'end': 1666727088.8754954,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_249.state')},
                       'cmd_25': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.output'),
                                  'state': {'begin': 1666726014.4914665,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.792306661605835,
                                            'end': 1666726015.2837732,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_25.state')},
                       'cmd_250': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.output'),
                                   'state': {'begin': 1666727090.9050484,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7911124229431152,
                                             'end': 1666727091.6961608,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_250.state')},
                       'cmd_251': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.output'),
                                   'state': {'begin': 1666727091.6984527,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.475045919418335,
                                             'end': 1666727103.1734986,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_251.state')},
                       'cmd_252': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734192,\n'
                                             '  "timestamp_ms" : '
                                             '1666734192584,\n'
                                             '  "time" : "Tue Oct 25 23:43:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6370635776,\n'
                                             '        "io_kbytes" : 6221324,\n'
                                             '        "bw_bytes" : 909960830,\n'
                                             '        "bw" : 888633,\n'
                                             '        "iops" : 222157.834595,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1555327,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1462,\n'
                                             '          "max" : 27873,\n'
                                             '          "mean" : 1640.823853,\n'
                                             '          "stddev" : '
                                             '204.088642,\n'
                                             '          "N" : 1555328\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 240,\n'
                                             '          "max" : 140985,\n'
                                             '          "mean" : '
                                             '16120.417763,\n'
                                             '          "stddev" : '
                                             '5496.166937,\n'
                                             '          "N" : 1555330,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14144,\n'
                                             '            "5.000000" : 14400,\n'
                                             '            "10.000000" : '
                                             '14528,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '14656,\n'
                                             '            "40.000000" : '
                                             '14656,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '14784,\n'
                                             '            "80.000000" : '
                                             '14912,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '36096,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '90624,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10810,\n'
                                             '          "max" : 142668,\n'
                                             '          "mean" : '
                                             '17761.241593,\n'
                                             '          "stddev" : '
                                             '5496.744254,\n'
                                             '          "N" : 1555330\n'
                                             '        },\n'
                                             '        "bw_min" : 887288,\n'
                                             '        "bw_max" : 890128,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '888840.714286,\n'
                                             '        "bw_dev" : 890.705375,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221822,\n'
                                             '        "iops_max" : 222532,\n'
                                             '        "iops_mean" : '
                                             '222210.285714,\n'
                                             '        "iops_stddev" : '
                                             '222.724193,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.785714,\n'
                                             '      "sys_cpu" : 81.185714,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.012152,\n'
                                             '        "20" : 91.790730,\n'
                                             '        "50" : 8.009377,\n'
                                             '        "100" : 0.164274,\n'
                                             '        "250" : 0.023082,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88037: Tue '
                                             'Oct 25 23:43:12 2022\n'
                                             '  read: IOPS=222k, BW=868MiB/s '
                                             '(910MB/s)(6076MiB/7001msec)\n'
                                             '    slat (nsec): min=1462, '
                                             'max=27873, avg=1640.82, '
                                             'stdev=204.09\n'
                                             '    clat (nsec): min=240, '
                                             'max=140985, avg=16120.42, '
                                             'stdev=5496.17\n'
                                             '     lat (usec): min=10, '
                                             'max=142, avg=17.76, stdev= 5.50\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   15], '
                                             '80.00th=[   15], 90.00th=[   '
                                             '17], 95.00th=[   28],\n'
                                             '     | 99.00th=[   37], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '80], 99.95th=[   91],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=887288, '
                                             'max=890128, per=100.00%, '
                                             'avg=888840.71, stdev=890.71, '
                                             'samples=14\n'
                                             '   iops        : min=221822, '
                                             'max=222532, avg=222210.29, '
                                             'stdev=222.72, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=91.79%, 50=8.01%, '
                                             '100=0.16%\n'
                                             '  lat (usec)   : 250=0.02%\n'
                                             '  cpu          : usr=18.79%, '
                                             'sys=81.19%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1555327,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=868MiB/s (910MB/s), '
                                             '868MiB/s-868MiB/s '
                                             '(910MB/s-910MB/s), io=6076MiB '
                                             '(6371MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.output'),
                                   'state': {'begin': 1666727103.1762161,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7970378398895264,
                                             'end': 1666727103.973254,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_252.state')},
                       'cmd_253': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.output'),
                                   'state': {'begin': 1666727104.959668,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.785696268081665,
                                             'end': 1666727105.7453642,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_253.state')},
                       'cmd_254': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.output'),
                                   'state': {'begin': 1666727105.752141,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.066343784332275,
                                             'end': 1666727116.8184848,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_254.state')},
                       'cmd_255': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734206,\n'
                                             '  "timestamp_ms" : '
                                             '1666734206301,\n'
                                             '  "time" : "Tue Oct 25 23:43:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4069163008,\n'
                                             '        "io_kbytes" : 3973792,\n'
                                             '        "bw_bytes" : 581225968,\n'
                                             '        "bw" : 567603,\n'
                                             '        "iops" : 141900.442794,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 993445,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2274,\n'
                                             '          "max" : 1343589,\n'
                                             '          "mean" : 2510.766837,\n'
                                             '          "stddev" : '
                                             '1415.574549,\n'
                                             '          "N" : 993445\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 240,\n'
                                             '          "max" : 1371542,\n'
                                             '          "mean" : '
                                             '25199.312395,\n'
                                             '          "stddev" : '
                                             '8781.217777,\n'
                                             '          "N" : 993448,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 14272,\n'
                                             '            "10.000000" : '
                                             '14400,\n'
                                             '            "20.000000" : '
                                             '14656,\n'
                                             '            "30.000000" : '
                                             '20864,\n'
                                             '            "40.000000" : '
                                             '21376,\n'
                                             '            "50.000000" : '
                                             '29568,\n'
                                             '            "60.000000" : '
                                             '30080,\n'
                                             '            "70.000000" : '
                                             '30080,\n'
                                             '            "80.000000" : '
                                             '30336,\n'
                                             '            "90.000000" : '
                                             '30592,\n'
                                             '            "95.000000" : '
                                             '35072,\n'
                                             '            "99.000000" : '
                                             '52992,\n'
                                             '            "99.500000" : '
                                             '56576,\n'
                                             '            "99.900000" : '
                                             '82432,\n'
                                             '            "99.950000" : '
                                             '97792,\n'
                                             '            "99.990000" : '
                                             '114176\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14237,\n'
                                             '          "max" : 1373906,\n'
                                             '          "mean" : '
                                             '27710.079092,\n'
                                             '          "stddev" : '
                                             '8975.141547,\n'
                                             '          "N" : 993448\n'
                                             '        },\n'
                                             '        "bw_min" : 566520,\n'
                                             '        "bw_max" : 570888,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '567735.857143,\n'
                                             '        "bw_dev" : 1371.074472,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 141630,\n'
                                             '        "iops_max" : 142722,\n'
                                             '        "iops_mean" : '
                                             '141933.928571,\n'
                                             '        "iops_stddev" : '
                                             '342.789118,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 20.185714,\n'
                                             '      "sys_cpu" : 56.685714,\n'
                                             '      "ctx" : 253738,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 24.128261,\n'
                                             '        "50" : 74.194948,\n'
                                             '        "100" : 1.629783,\n'
                                             '        "250" : 0.044089,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88076: Tue '
                                             'Oct 25 23:43:26 2022\n'
                                             '  read: IOPS=142k, BW=554MiB/s '
                                             '(581MB/s)(3881MiB/7001msec)\n'
                                             '    slat (usec): min=2, '
                                             'max=1343, avg= 2.51, stdev= '
                                             '1.42\n'
                                             '    clat (nsec): min=240, '
                                             'max=1371.5k, avg=25199.31, '
                                             'stdev=8781.22\n'
                                             '     lat (usec): min=14, '
                                             'max=1373, avg=27.71, stdev= '
                                             '8.98\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   21], '
                                             '40.00th=[   22], 50.00th=[   '
                                             '30], 60.00th=[   31],\n'
                                             '     | 70.00th=[   31], '
                                             '80.00th=[   31], 90.00th=[   '
                                             '31], 95.00th=[   36],\n'
                                             '     | 99.00th=[   53], '
                                             '99.50th=[   57], 99.90th=[   '
                                             '83], 99.95th=[   98],\n'
                                             '     | 99.99th=[  115]\n'
                                             '   bw (  KiB/s): min=566520, '
                                             'max=570888, per=100.00%, '
                                             'avg=567735.86, stdev=1371.07, '
                                             'samples=14\n'
                                             '   iops        : min=141630, '
                                             'max=142722, avg=141933.93, '
                                             'stdev=342.79, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=24.13%, 50=74.19%, '
                                             '100=1.63%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=20.19%, '
                                             'sys=56.69%, ctx=253738, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=993445,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=554MiB/s (581MB/s), '
                                             '554MiB/s-554MiB/s '
                                             '(581MB/s-581MB/s), io=3881MiB '
                                             '(4069MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.output'),
                                   'state': {'begin': 1666727116.8215501,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8406717777252197,
                                             'end': 1666727117.662222,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_255.state')},
                       'cmd_256': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.output'),
                                   'state': {'begin': 1666727118.686377,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7901842594146729,
                                             'end': 1666727119.4765613,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_256.state')},
                       'cmd_257': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.output'),
                                   'state': {'begin': 1666727119.4785686,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.056126117706299,
                                             'end': 1666727130.5346947,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_257.state')},
                       'cmd_258': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734220,\n'
                                             '  "timestamp_ms" : '
                                             '1666734220017,\n'
                                             '  "time" : "Tue Oct 25 23:43:40 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4071936000,\n'
                                             '        "io_kbytes" : 3976500,\n'
                                             '        "bw_bytes" : 581622053,\n'
                                             '        "bw" : 567990,\n'
                                             '        "iops" : 141997.000429,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 994121,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2275,\n'
                                             '          "max" : 47789,\n'
                                             '          "mean" : 2511.122108,\n'
                                             '          "stddev" : '
                                             '454.419258,\n'
                                             '          "N" : 994122\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 271,\n'
                                             '          "max" : 165681,\n'
                                             '          "mean" : '
                                             '25181.048494,\n'
                                             '          "stddev" : '
                                             '8477.785485,\n'
                                             '          "N" : 994124,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 14144,\n'
                                             '            "10.000000" : '
                                             '14272,\n'
                                             '            "20.000000" : '
                                             '14656,\n'
                                             '            "30.000000" : '
                                             '20864,\n'
                                             '            "40.000000" : '
                                             '21376,\n'
                                             '            "50.000000" : '
                                             '29568,\n'
                                             '            "60.000000" : '
                                             '30080,\n'
                                             '            "70.000000" : '
                                             '30080,\n'
                                             '            "80.000000" : '
                                             '30336,\n'
                                             '            "90.000000" : '
                                             '30592,\n'
                                             '            "95.000000" : '
                                             '34560,\n'
                                             '            "99.000000" : '
                                             '52992,\n'
                                             '            "99.500000" : '
                                             '56576,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '96768,\n'
                                             '            "99.990000" : '
                                             '115200\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11572,\n'
                                             '          "max" : 168056,\n'
                                             '          "mean" : '
                                             '27692.170811,\n'
                                             '          "stddev" : '
                                             '8578.284311,\n'
                                             '          "N" : 994124\n'
                                             '        },\n'
                                             '        "bw_min" : 565416,\n'
                                             '        "bw_max" : 571520,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '568149.071429,\n'
                                             '        "bw_dev" : 1825.775977,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 141354,\n'
                                             '        "iops_max" : 142880,\n'
                                             '        "iops_mean" : '
                                             '142037.214286,\n'
                                             '        "iops_stddev" : '
                                             '456.460661,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 18.328571,\n'
                                             '      "sys_cpu" : 58.714286,\n'
                                             '      "ctx" : 253915,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 24.695384,\n'
                                             '        "50" : 73.607136,\n'
                                             '        "100" : 1.651710,\n'
                                             '        "250" : 0.043355,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88116: Tue '
                                             'Oct 25 23:43:40 2022\n'
                                             '  read: IOPS=142k, BW=555MiB/s '
                                             '(582MB/s)(3883MiB/7001msec)\n'
                                             '    slat (nsec): min=2275, '
                                             'max=47789, avg=2511.12, '
                                             'stdev=454.42\n'
                                             '    clat (nsec): min=271, '
                                             'max=165681, avg=25181.05, '
                                             'stdev=8477.79\n'
                                             '     lat (usec): min=11, '
                                             'max=168, avg=27.69, stdev= 8.58\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   21], '
                                             '40.00th=[   22], 50.00th=[   '
                                             '30], 60.00th=[   31],\n'
                                             '     | 70.00th=[   31], '
                                             '80.00th=[   31], 90.00th=[   '
                                             '31], 95.00th=[   35],\n'
                                             '     | 99.00th=[   53], '
                                             '99.50th=[   57], 99.90th=[   '
                                             '82], 99.95th=[   97],\n'
                                             '     | 99.99th=[  116]\n'
                                             '   bw (  KiB/s): min=565416, '
                                             'max=571520, per=100.00%, '
                                             'avg=568149.07, stdev=1825.78, '
                                             'samples=14\n'
                                             '   iops        : min=141354, '
                                             'max=142880, avg=142037.21, '
                                             'stdev=456.46, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=24.70%, 50=73.61%, '
                                             '100=1.65%\n'
                                             '  lat (usec)   : 250=0.04%\n'
                                             '  cpu          : usr=18.33%, '
                                             'sys=58.71%, ctx=253915, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=994121,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=555MiB/s (582MB/s), '
                                             '555MiB/s-555MiB/s '
                                             '(582MB/s-582MB/s), io=3883MiB '
                                             '(4072MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.output'),
                                   'state': {'begin': 1666727130.5379033,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8043882846832275,
                                             'end': 1666727131.3422916,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_258.state')},
                       'cmd_259': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.output'),
                                   'state': {'begin': 1666727132.333408,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7911653518676758,
                                             'end': 1666727133.1245735,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_259.state')},
                       'cmd_26': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.output'),
                                  'state': {'begin': 1666726015.2861967,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring_cmd-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="0" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.375238418579102,
                                            'end': 1666726026.6614351,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_26.state')},
                       'cmd_260': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.output'),
                                   'state': {'begin': 1666727133.127158,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_io_uring-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.059845924377441,
                                             'end': 1666727144.1870039,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_260.state')},
                       'cmd_261': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734233,\n'
                                             '  "timestamp_ms" : '
                                             '1666734233669,\n'
                                             '  "time" : "Tue Oct 25 23:43:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '4061618176,\n'
                                             '        "io_kbytes" : 3966424,\n'
                                             '        "bw_bytes" : 580148289,\n'
                                             '        "bw" : 566551,\n'
                                             '        "iops" : 141637.194686,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 991602,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 2294,\n'
                                             '          "max" : 197260,\n'
                                             '          "mean" : 2527.106294,\n'
                                             '          "stddev" : '
                                             '495.502687,\n'
                                             '          "N" : 991603\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 230,\n'
                                             '          "max" : 212398,\n'
                                             '          "mean" : '
                                             '25205.674751,\n'
                                             '          "stddev" : '
                                             '8552.840582,\n'
                                             '          "N" : 991605,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 14144,\n'
                                             '            "10.000000" : '
                                             '14272,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '20864,\n'
                                             '            "40.000000" : '
                                             '21120,\n'
                                             '            "50.000000" : '
                                             '29824,\n'
                                             '            "60.000000" : '
                                             '30080,\n'
                                             '            "70.000000" : '
                                             '30336,\n'
                                             '            "80.000000" : '
                                             '30336,\n'
                                             '            "90.000000" : '
                                             '30592,\n'
                                             '            "95.000000" : '
                                             '35072,\n'
                                             '            "99.000000" : '
                                             '52480,\n'
                                             '            "99.500000" : '
                                             '55552,\n'
                                             '            "99.900000" : '
                                             '84480,\n'
                                             '            "99.950000" : '
                                             '98816,\n'
                                             '            "99.990000" : '
                                             '116224\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11661,\n'
                                             '          "max" : 215053,\n'
                                             '          "mean" : '
                                             '27732.780705,\n'
                                             '          "stddev" : '
                                             '8653.005043,\n'
                                             '          "N" : 991605\n'
                                             '        },\n'
                                             '        "bw_min" : 565176,\n'
                                             '        "bw_max" : 569450,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '566683.571429,\n'
                                             '        "bw_dev" : 1091.546871,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 141294,\n'
                                             '        "iops_max" : 142362,\n'
                                             '        "iops_mean" : '
                                             '141670.857143,\n'
                                             '        "iops_stddev" : '
                                             '272.789256,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.257143,\n'
                                             '      "sys_cpu" : 55.942857,\n'
                                             '      "ctx" : 253051,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 23.960924,\n'
                                             '        "50" : 74.303501,\n'
                                             '        "100" : 1.684345,\n'
                                             '        "250" : 0.046793,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88156: Tue '
                                             'Oct 25 23:43:53 2022\n'
                                             '  read: IOPS=142k, BW=553MiB/s '
                                             '(580MB/s)(3873MiB/7001msec)\n'
                                             '    slat (nsec): min=2294, '
                                             'max=197260, avg=2527.11, '
                                             'stdev=495.50\n'
                                             '    clat (nsec): min=230, '
                                             'max=212398, avg=25205.67, '
                                             'stdev=8552.84\n'
                                             '     lat (usec): min=11, '
                                             'max=215, avg=27.73, stdev= 8.65\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   21], '
                                             '40.00th=[   22], 50.00th=[   '
                                             '30], 60.00th=[   31],\n'
                                             '     | 70.00th=[   31], '
                                             '80.00th=[   31], 90.00th=[   '
                                             '31], 95.00th=[   36],\n'
                                             '     | 99.00th=[   53], '
                                             '99.50th=[   56], 99.90th=[   '
                                             '85], 99.95th=[   99],\n'
                                             '     | 99.99th=[  117]\n'
                                             '   bw (  KiB/s): min=565176, '
                                             'max=569450, per=100.00%, '
                                             'avg=566683.57, stdev=1091.55, '
                                             'samples=14\n'
                                             '   iops        : min=141294, '
                                             'max=142362, avg=141670.86, '
                                             'stdev=272.79, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%, '
                                             '500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=23.96%, 50=74.30%, '
                                             '100=1.68%\n'
                                             '  lat (usec)   : 250=0.05%\n'
                                             '  cpu          : usr=21.26%, '
                                             'sys=55.94%, ctx=253051, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=991602,0,0,0 short=0,0,0,0 '
                                             'dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=553MiB/s (580MB/s), '
                                             '553MiB/s-553MiB/s '
                                             '(580MB/s-580MB/s), io=3873MiB '
                                             '(4062MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.output'),
                                   'state': {'begin': 1666727144.19009,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7823362350463867,
                                             'end': 1666727144.9724262,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_261.state')},
                       'cmd_262': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.output'),
                                   'state': {'begin': 1666727145.9689424,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.784095048904419,
                                             'end': 1666727146.7530375,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_262.state')},
                       'cmd_263': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.output'),
                                   'state': {'begin': 1666727146.7549021,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.070714235305786,
                                             'end': 1666727157.8256164,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_263.state')},
                       'cmd_264': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734247,\n'
                                             '  "timestamp_ms" : '
                                             '1666734247308,\n'
                                             '  "time" : "Tue Oct 25 23:44:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5208461312,\n'
                                             '        "io_kbytes" : 5086388,\n'
                                             '        "bw_bytes" : 743959621,\n'
                                             '        "bw" : 726523,\n'
                                             '        "iops" : 181630.195686,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1271593,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1994,\n'
                                             '          "max" : 27552,\n'
                                             '          "mean" : 2273.459399,\n'
                                             '          "stddev" : '
                                             '184.930489,\n'
                                             '          "N" : 1271594\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 6773,\n'
                                             '          "max" : 133701,\n'
                                             '          "mean" : '
                                             '19356.888477,\n'
                                             '          "stddev" : '
                                             '6100.168271,\n'
                                             '          "N" : 1271596,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14400,\n'
                                             '            "5.000000" : 14528,\n'
                                             '            "10.000000" : '
                                             '14656,\n'
                                             '            "20.000000" : '
                                             '14784,\n'
                                             '            "30.000000" : '
                                             '15168,\n'
                                             '            "40.000000" : '
                                             '15424,\n'
                                             '            "50.000000" : '
                                             '18560,\n'
                                             '            "60.000000" : '
                                             '21120,\n'
                                             '            "70.000000" : '
                                             '21120,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '21888,\n'
                                             '            "95.000000" : '
                                             '30848,\n'
                                             '            "99.000000" : '
                                             '42240,\n'
                                             '            "99.500000" : '
                                             '47360,\n'
                                             '            "99.900000" : '
                                             '77312,\n'
                                             '            "99.950000" : '
                                             '91648,\n'
                                             '            "99.990000" : '
                                             '104960\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 11512,\n'
                                             '          "max" : 135855,\n'
                                             '          "mean" : '
                                             '21630.347846,\n'
                                             '          "stddev" : '
                                             '6127.066883,\n'
                                             '          "N" : 1271596\n'
                                             '        },\n'
                                             '        "bw_min" : 724216,\n'
                                             '        "bw_max" : 728976,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '726728.285714,\n'
                                             '        "bw_dev" : 1331.121877,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 181054,\n'
                                             '        "iops_max" : 182244,\n'
                                             '        "iops_mean" : '
                                             '181682.071429,\n'
                                             '        "iops_stddev" : '
                                             '332.780469,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.585714,\n'
                                             '      "sys_cpu" : 48.000000,\n'
                                             '      "ctx" : 330200,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 50.625082,\n'
                                             '        "50" : 49.125703,\n'
                                             '        "100" : 0.219253,\n'
                                             '        "250" : 0.023750,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1806255,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 26402,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 26402,\n'
                                             '      "util" : 99.078934\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=88195: Tue '
                                             'Oct 25 23:44:07 2022\n'
                                             '  read: IOPS=182k, BW=709MiB/s '
                                             '(744MB/s)(4967MiB/7001msec)\n'
                                             '    slat (nsec): min=1994, '
                                             'max=27552, avg=2273.46, '
                                             'stdev=184.93\n'
                                             '    clat (usec): min=6, max=133, '
                                             'avg=19.36, stdev= 6.10\n'
                                             '     lat (usec): min=11, '
                                             'max=135, avg=21.63, stdev= 6.13\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   16], '
                                             '40.00th=[   16], 50.00th=[   '
                                             '19], 60.00th=[   22],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '22], 95.00th=[   31],\n'
                                             '     | 99.00th=[   43], '
                                             '99.50th=[   48], 99.90th=[   '
                                             '78], 99.95th=[   92],\n'
                                             '     | 99.99th=[  105]\n'
                                             '   bw (  KiB/s): min=724216, '
                                             'max=728976, per=100.00%, '
                                             'avg=726728.29, stdev=1331.12, '
                                             'samples=14\n'
                                             '   iops        : min=181054, '
                                             'max=182244, avg=181682.07, '
                                             'stdev=332.78, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=50.63%, 50=49.13%, 100=0.22%, '
                                             '250=0.02%\n'
                                             '  cpu          : usr=21.59%, '
                                             'sys=48.00%, ctx=330200, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1271593,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=709MiB/s (744MB/s), '
                                             '709MiB/s-709MiB/s '
                                             '(744MB/s-744MB/s), io=4967MiB '
                                             '(5208MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1806255/0, '
                                             'merge=0/0, ticks=26402/0, '
                                             'in_queue=26402, util=99.08%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.output'),
                                   'state': {'begin': 1666727157.8282857,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7942020893096924,
                                             'end': 1666727158.6224878,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_264.state')},
                       'cmd_265': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.output'),
                                   'state': {'begin': 1666727159.6133997,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7875704765319824,
                                             'end': 1666727160.4009702,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_265.state')},
                       'cmd_266': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.output'),
                                   'state': {'begin': 1666727160.403202,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.080987930297852,
                                             'end': 1666727171.48419,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_266.state')},
                       'cmd_267': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734260,\n'
                                             '  "timestamp_ms" : '
                                             '1666734260967,\n'
                                             '  "time" : "Tue Oct 25 23:44:20 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5129441280,\n'
                                             '        "io_kbytes" : 5009220,\n'
                                             '        "bw_bytes" : 732672658,\n'
                                             '        "bw" : 715500,\n'
                                             '        "iops" : 178874.589344,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1252301,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1342,\n'
                                             '          "max" : 38192,\n'
                                             '          "mean" : 2214.863789,\n'
                                             '          "stddev" : '
                                             '1428.389632,\n'
                                             '          "N" : 1252302\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2564,\n'
                                             '          "max" : 191189,\n'
                                             '          "mean" : '
                                             '19851.901731,\n'
                                             '          "stddev" : '
                                             '5831.909239,\n'
                                             '          "N" : 1252304,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 15040,\n'
                                             '            "10.000000" : '
                                             '15552,\n'
                                             '            "20.000000" : '
                                             '15808,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '19072,\n'
                                             '            "60.000000" : '
                                             '19584,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '21888,\n'
                                             '            "95.000000" : '
                                             '31616,\n'
                                             '            "99.000000" : '
                                             '43264,\n'
                                             '            "99.500000" : '
                                             '46336,\n'
                                             '            "99.900000" : '
                                             '78336,\n'
                                             '            "99.950000" : '
                                             '92672,\n'
                                             '            "99.990000" : '
                                             '104960\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10289,\n'
                                             '          "max" : 195036,\n'
                                             '          "mean" : '
                                             '22066.766055,\n'
                                             '          "stddev" : '
                                             '5810.190816,\n'
                                             '          "N" : 1252304\n'
                                             '        },\n'
                                             '        "bw_min" : 705904,\n'
                                             '        "bw_max" : 720104,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '715783.785714,\n'
                                             '        "bw_dev" : 3968.759430,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 176476,\n'
                                             '        "iops_max" : 180026,\n'
                                             '        "iops_mean" : '
                                             '178945.857143,\n'
                                             '        "iops_stddev" : '
                                             '992.094820,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.257143,\n'
                                             '      "sys_cpu" : 45.128571,\n'
                                             '      "ctx" : 319706,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.063084,\n'
                                             '        "20" : 61.387717,\n'
                                             '        "50" : 38.299898,\n'
                                             '        "100" : 0.221991,\n'
                                             '        "250" : 0.026990,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1786396,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 28771,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 28771,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=88233: Tue '
                                             'Oct 25 23:44:20 2022\n'
                                             '  read: IOPS=179k, BW=699MiB/s '
                                             '(733MB/s)(4892MiB/7001msec)\n'
                                             '    slat (nsec): min=1342, '
                                             'max=38192, avg=2214.86, '
                                             'stdev=1428.39\n'
                                             '    clat (usec): min=2, max=191, '
                                             'avg=19.85, stdev= 5.83\n'
                                             '     lat (usec): min=10, '
                                             'max=195, avg=22.07, stdev= 5.81\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   16], 10.00th=[   16], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '20], 60.00th=[   20],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '22], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   47], 99.90th=[   '
                                             '79], 99.95th=[   93],\n'
                                             '     | 99.99th=[  105]\n'
                                             '   bw (  KiB/s): min=705904, '
                                             'max=720104, per=100.00%, '
                                             'avg=715783.79, stdev=3968.76, '
                                             'samples=14\n'
                                             '   iops        : min=176476, '
                                             'max=180026, avg=178945.86, '
                                             'stdev=992.09, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.06%, 20=61.39%, 50=38.30%, '
                                             '100=0.22%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=17.26%, '
                                             'sys=45.13%, ctx=319706, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1252301,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=699MiB/s (733MB/s), '
                                             '699MiB/s-699MiB/s '
                                             '(733MB/s-733MB/s), io=4892MiB '
                                             '(5129MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1786396/0, '
                                             'merge=0/0, ticks=28771/0, '
                                             'in_queue=28771, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.output'),
                                   'state': {'begin': 1666727171.4907925,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7877211570739746,
                                             'end': 1666727172.2785137,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_267.state')},
                       'cmd_268': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.output'),
                                   'state': {'begin': 1666727173.265863,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7897942066192627,
                                             'end': 1666727174.0556571,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_268.state')},
                       'cmd_269': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.output'),
                                   'state': {'begin': 1666727174.0578525,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.066452026367188,
                                             'end': 1666727185.1243045,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_269.state')},
                       'cmd_27': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733116,\n'
                                            '  "timestamp_ms" : '
                                            '1666733116091,\n'
                                            '  "time" : "Tue Oct 25 23:25:16 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring_cmd-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "0",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2407424000,\n'
                                            '        "io_kbytes" : 2351000,\n'
                                            '        "bw_bytes" : 343868590,\n'
                                            '        "bw" : 335809,\n'
                                            '        "iops" : 83952.292530,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 587750,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 711,\n'
                                            '          "max" : 11492,\n'
                                            '          "mean" : 925.037659,\n'
                                            '          "stddev" : 79.223115,\n'
                                            '          "N" : 587750\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 3837,\n'
                                            '          "max" : 176060,\n'
                                            '          "mean" : 10698.287205,\n'
                                            '          "stddev" : '
                                            '2205.678474,\n'
                                            '          "N" : 587750,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9920,\n'
                                            '            "5.000000" : 10048,\n'
                                            '            "10.000000" : 10048,\n'
                                            '            "20.000000" : 10176,\n'
                                            '            "30.000000" : 10304,\n'
                                            '            "40.000000" : 10304,\n'
                                            '            "50.000000" : 10432,\n'
                                            '            "60.000000" : 10432,\n'
                                            '            "70.000000" : 10560,\n'
                                            '            "80.000000" : 10560,\n'
                                            '            "90.000000" : 10688,\n'
                                            '            "95.000000" : 11072,\n'
                                            '            "99.000000" : 19840,\n'
                                            '            "99.500000" : 27520,\n'
                                            '            "99.900000" : 28032,\n'
                                            '            "99.950000" : 43776,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 10570,\n'
                                            '          "max" : 179687,\n'
                                            '          "mean" : 11623.324864,\n'
                                            '          "stddev" : '
                                            '2208.087793,\n'
                                            '          "N" : 587750\n'
                                            '        },\n'
                                            '        "bw_min" : 335496,\n'
                                            '        "bw_max" : 336312,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '335897.642857,\n'
                                            '        "bw_dev" : 239.573213,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83874,\n'
                                            '        "iops_max" : 84078,\n'
                                            '        "iops_mean" : '
                                            '83974.357143,\n'
                                            '        "iops_stddev" : '
                                            '59.925091,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 99.985714,\n'
                                            '      "sys_cpu" : 0.000000,\n'
                                            '      "ctx" : 20,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 3.303956,\n'
                                            '        "20" : 95.712123,\n'
                                            '        "50" : 0.938494,\n'
                                            '        "100" : 0.044747,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring_cmd-bdev_xnvme: '
                                            '(groupid=0, jobs=1): err= 0: '
                                            'pid=84733: Tue Oct 25 23:25:16 '
                                            '2022\n'
                                            '  read: IOPS=84.0k, BW=328MiB/s '
                                            '(344MB/s)(2296MiB/7001msec)\n'
                                            '    slat (nsec): min=711, '
                                            'max=11492, avg=925.04, '
                                            'stdev=79.22\n'
                                            '    clat (usec): min=3, max=176, '
                                            'avg=10.70, stdev= 2.21\n'
                                            '     lat (usec): min=10, max=179, '
                                            'avg=11.62, stdev= 2.21\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9920],  '
                                            '5.00th=[10048], 10.00th=[10048], '
                                            '20.00th=[10176],\n'
                                            '     | 30.00th=[10304], '
                                            '40.00th=[10304], 50.00th=[10432], '
                                            '60.00th=[10432],\n'
                                            '     | 70.00th=[10560], '
                                            '80.00th=[10560], 90.00th=[10688], '
                                            '95.00th=[11072],\n'
                                            '     | 99.00th=[19840], '
                                            '99.50th=[27520], 99.90th=[28032], '
                                            '99.95th=[43776],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=335496, '
                                            'max=336312, per=100.00%, '
                                            'avg=335897.64, stdev=239.57, '
                                            'samples=14\n'
                                            '   iops        : min=83874, '
                                            'max=84078, avg=83974.36, '
                                            'stdev=59.93, samples=14\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=3.30%, 20=95.71%, 50=0.94%, '
                                            '100=0.04%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=99.99%, '
                                            'sys=0.00%, ctx=20, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=587750,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=328MiB/s (344MB/s), '
                                            '328MiB/s-328MiB/s '
                                            '(344MB/s-344MB/s), io=2296MiB '
                                            '(2407MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.output'),
                                  'state': {'begin': 1666726026.6640089,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7969875335693359,
                                            'end': 1666726027.4609964,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_27.state')},
                       'cmd_270': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734274,\n'
                                             '  "timestamp_ms" : '
                                             '1666734274606,\n'
                                             '  "time" : "Tue Oct 25 23:44:34 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5094621184,\n'
                                             '        "io_kbytes" : 4975216,\n'
                                             '        "bw_bytes" : 727699069,\n'
                                             '        "bw" : 710643,\n'
                                             '        "iops" : 177660.477075,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1243801,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1322,\n'
                                             '          "max" : 179357,\n'
                                             '          "mean" : 2270.165353,\n'
                                             '          "stddev" : '
                                             '1482.663537,\n'
                                             '          "N" : 1243801\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2415,\n'
                                             '          "max" : 1314775,\n'
                                             '          "mean" : '
                                             '19921.951998,\n'
                                             '          "stddev" : '
                                             '6421.548737,\n'
                                             '          "N" : 1243804,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14016,\n'
                                             '            "5.000000" : 15040,\n'
                                             '            "10.000000" : '
                                             '15552,\n'
                                             '            "20.000000" : '
                                             '15808,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '19072,\n'
                                             '            "60.000000" : '
                                             '19328,\n'
                                             '            "70.000000" : '
                                             '21120,\n'
                                             '            "80.000000" : '
                                             '21376,\n'
                                             '            "90.000000" : '
                                             '23680,\n'
                                             '            "95.000000" : '
                                             '31872,\n'
                                             '            "99.000000" : '
                                             '43264,\n'
                                             '            "99.500000" : '
                                             '47360,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '93696,\n'
                                             '            "99.990000" : '
                                             '104960\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9989,\n'
                                             '          "max" : 1316138,\n'
                                             '          "mean" : '
                                             '22192.118014,\n'
                                             '          "stddev" : '
                                             '6367.348917,\n'
                                             '          "N" : 1243804\n'
                                             '        },\n'
                                             '        "bw_min" : 653496,\n'
                                             '        "bw_max" : 722680,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '710909.000000,\n'
                                             '        "bw_dev" : '
                                             '18576.458195,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 163374,\n'
                                             '        "iops_max" : 180670,\n'
                                             '        "iops_mean" : '
                                             '177727.285714,\n'
                                             '        "iops_stddev" : '
                                             '4644.186298,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.771429,\n'
                                             '      "sys_cpu" : 45.885714,\n'
                                             '      "ctx" : 313922,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.049365,\n'
                                             '        "20" : 62.858367,\n'
                                             '        "50" : 36.778311,\n'
                                             '        "100" : 0.285978,\n'
                                             '        "250" : 0.027336,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 1774792,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 28485,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 28485,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=88271: Tue '
                                             'Oct 25 23:44:34 2022\n'
                                             '  read: IOPS=178k, BW=694MiB/s '
                                             '(728MB/s)(4859MiB/7001msec)\n'
                                             '    slat (nsec): min=1322, '
                                             'max=179357, avg=2270.17, '
                                             'stdev=1482.66\n'
                                             '    clat (usec): min=2, '
                                             'max=1314, avg=19.92, stdev= '
                                             '6.42\n'
                                             '     lat (usec): min=9, '
                                             'max=1316, avg=22.19, stdev= '
                                             '6.37\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   16], 10.00th=[   16], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '20], 60.00th=[   20],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '24], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   48], 99.90th=[   '
                                             '80], 99.95th=[   94],\n'
                                             '     | 99.99th=[  105]\n'
                                             '   bw (  KiB/s): min=653496, '
                                             'max=722680, per=100.00%, '
                                             'avg=710909.00, stdev=18576.46, '
                                             'samples=14\n'
                                             '   iops        : min=163374, '
                                             'max=180670, avg=177727.29, '
                                             'stdev=4644.19, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.05%, 20=62.86%, 50=36.78%, '
                                             '100=0.29%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=17.77%, '
                                             'sys=45.89%, ctx=313922, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1243801,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=694MiB/s (728MB/s), '
                                             '694MiB/s-694MiB/s '
                                             '(728MB/s-728MB/s), io=4859MiB '
                                             '(5095MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=1774792/0, '
                                             'merge=0/0, ticks=28485/0, '
                                             'in_queue=28485, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.output'),
                                   'state': {'begin': 1666727185.1269662,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8012464046478271,
                                             'end': 1666727185.9282126,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_270.state')},
                       'cmd_271': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.output'),
                                   'state': {'begin': 1666727187.9529927,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.787787675857544,
                                             'end': 1666727188.7407804,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_271.state')},
                       'cmd_272': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.output'),
                                   'state': {'begin': 1666727188.742865,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.424794912338257,
                                             'end': 1666727200.16766,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_272.state')},
                       'cmd_273': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734289,\n'
                                             '  "timestamp_ms" : '
                                             '1666734289535,\n'
                                             '  "time" : "Tue Oct 25 23:44:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6368309248,\n'
                                             '        "io_kbytes" : 6219052,\n'
                                             '        "bw_bytes" : 909628517,\n'
                                             '        "bw" : 888309,\n'
                                             '        "iops" : 222076.846165,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1554760,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1523,\n'
                                             '          "max" : 20198,\n'
                                             '          "mean" : 1636.536629,\n'
                                             '          "stddev" : '
                                             '223.818184,\n'
                                             '          "N" : 1554760\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 251,\n'
                                             '          "max" : 148058,\n'
                                             '          "mean" : '
                                             '16084.288581,\n'
                                             '          "stddev" : '
                                             '5476.140778,\n'
                                             '          "N" : 1554763,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13504,\n'
                                             '            "5.000000" : 14400,\n'
                                             '            "10.000000" : '
                                             '14528,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '14656,\n'
                                             '            "40.000000" : '
                                             '14656,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '14784,\n'
                                             '            "80.000000" : '
                                             '14912,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '36096,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '90624,\n'
                                             '            "99.990000" : '
                                             '104960\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9618,\n'
                                             '          "max" : 149671,\n'
                                             '          "mean" : '
                                             '17720.825126,\n'
                                             '          "stddev" : '
                                             '5474.918573,\n'
                                             '          "N" : 1554763\n'
                                             '        },\n'
                                             '        "bw_min" : 886712,\n'
                                             '        "bw_max" : 890216,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '888525.785714,\n'
                                             '        "bw_dev" : 1043.348321,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221678,\n'
                                             '        "iops_max" : 222554,\n'
                                             '        "iops_mean" : '
                                             '222131.428571,\n'
                                             '        "iops_stddev" : '
                                             '260.832311,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 73.700000,\n'
                                             '      "sys_cpu" : 26.285714,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.372984,\n'
                                             '        "2" : 0.960470,\n'
                                             '        "4" : 98.666547,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.063933,\n'
                                             '        "20" : 92.064048,\n'
                                             '        "50" : 7.686588,\n'
                                             '        "100" : 0.157516,\n'
                                             '        "250" : 0.025663,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=88319: Tue '
                                             'Oct 25 23:44:49 2022\n'
                                             '  read: IOPS=222k, BW=867MiB/s '
                                             '(910MB/s)(6073MiB/7001msec)\n'
                                             '    slat (nsec): min=1523, '
                                             'max=20198, avg=1636.54, '
                                             'stdev=223.82\n'
                                             '    clat (nsec): min=251, '
                                             'max=148058, avg=16084.29, '
                                             'stdev=5476.14\n'
                                             '     lat (usec): min=9, max=149, '
                                             'avg=17.72, stdev= 5.47\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   14],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   15], '
                                             '80.00th=[   15], 90.00th=[   '
                                             '17], 95.00th=[   28],\n'
                                             '     | 99.00th=[   37], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '80], 99.95th=[   91],\n'
                                             '     | 99.99th=[  105]\n'
                                             '   bw (  KiB/s): min=886712, '
                                             'max=890216, per=100.00%, '
                                             'avg=888525.79, stdev=1043.35, '
                                             'samples=14\n'
                                             '   iops        : min=221678, '
                                             'max=222554, avg=222131.43, '
                                             'stdev=260.83, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.06%, 20=92.06%, 50=7.69%, '
                                             '100=0.16%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=73.70%, '
                                             'sys=26.29%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, 2=1.0%, '
                                             '4=98.7%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1554760,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=867MiB/s (910MB/s), '
                                             '867MiB/s-867MiB/s '
                                             '(910MB/s-910MB/s), io=6073MiB '
                                             '(6368MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.output'),
                                   'state': {'begin': 1666727200.1704807,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7871673107147217,
                                             'end': 1666727200.957648,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_273.state')},
                       'cmd_274': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.output'),
                                   'state': {'begin': 1666727202.9961932,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.801347017288208,
                                             'end': 1666727203.7975402,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_274.state')},
                       'cmd_275': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.output'),
                                   'state': {'begin': 1666727203.80041,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.53487515449524,
                                             'end': 1666727215.3352852,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_275.state')},
                       'cmd_276': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734304,\n'
                                             '  "timestamp_ms" : '
                                             '1666734304702,\n'
                                             '  "time" : "Tue Oct 25 23:45:04 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6271488000,\n'
                                             '        "io_kbytes" : 6124500,\n'
                                             '        "bw_bytes" : 895798885,\n'
                                             '        "bw" : 874803,\n'
                                             '        "iops" : 218700.328524,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1531121,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1643,\n'
                                             '          "max" : 62267,\n'
                                             '          "mean" : 1883.836522,\n'
                                             '          "stddev" : '
                                             '167.538851,\n'
                                             '          "N" : 1531122\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 290,\n'
                                             '          "max" : 141535,\n'
                                             '          "mean" : '
                                             '16162.363481,\n'
                                             '          "stddev" : '
                                             '6497.223878,\n'
                                             '          "N" : 1531121,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10432,\n'
                                             '            "5.000000" : 10432,\n'
                                             '            "10.000000" : '
                                             '10560,\n'
                                             '            "20.000000" : '
                                             '12480,\n'
                                             '            "30.000000" : '
                                             '12864,\n'
                                             '            "40.000000" : '
                                             '14272,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '17024,\n'
                                             '            "80.000000" : '
                                             '19328,\n'
                                             '            "90.000000" : '
                                             '21376,\n'
                                             '            "95.000000" : '
                                             '28800,\n'
                                             '            "99.000000" : '
                                             '41216,\n'
                                             '            "99.500000" : '
                                             '44800,\n'
                                             '            "99.900000" : '
                                             '77312,\n'
                                             '            "99.950000" : '
                                             '94720,\n'
                                             '            "99.990000" : '
                                             '121344\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9808,\n'
                                             '          "max" : 143308,\n'
                                             '          "mean" : '
                                             '18046.199945,\n'
                                             '          "stddev" : '
                                             '6499.210762,\n'
                                             '          "N" : 1531121\n'
                                             '        },\n'
                                             '        "bw_min" : 873272,\n'
                                             '        "bw_max" : 877680,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '875229.571429,\n'
                                             '        "bw_dev" : 1466.632654,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 218318,\n'
                                             '        "iops_max" : 219420,\n'
                                             '        "iops_mean" : '
                                             '218807.285714,\n'
                                             '        "iops_stddev" : '
                                             '366.406758,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 77.814286,\n'
                                             '      "sys_cpu" : 22.157143,\n'
                                             '      "ctx" : 22,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.441180,\n'
                                             '        "4" : 99.548239,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.063418,\n'
                                             '        "20" : 80.184910,\n'
                                             '        "50" : 19.526347,\n'
                                             '        "100" : 0.182611,\n'
                                             '        "250" : 0.041016,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=88368: Tue '
                                             'Oct 25 23:45:04 2022\n'
                                             '  read: IOPS=219k, BW=854MiB/s '
                                             '(896MB/s)(5981MiB/7001msec)\n'
                                             '    slat (nsec): min=1643, '
                                             'max=62267, avg=1883.84, '
                                             'stdev=167.54\n'
                                             '    clat (nsec): min=290, '
                                             'max=141535, avg=16162.36, '
                                             'stdev=6497.22\n'
                                             '     lat (usec): min=9, max=143, '
                                             'avg=18.05, stdev= 6.50\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   11],  '
                                             '5.00th=[   11], 10.00th=[   11], '
                                             '20.00th=[   13],\n'
                                             '     | 30.00th=[   13], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   18], '
                                             '80.00th=[   20], 90.00th=[   '
                                             '22], 95.00th=[   29],\n'
                                             '     | 99.00th=[   42], '
                                             '99.50th=[   45], 99.90th=[   '
                                             '78], 99.95th=[   95],\n'
                                             '     | 99.99th=[  122]\n'
                                             '   bw (  KiB/s): min=873272, '
                                             'max=877680, per=100.00%, '
                                             'avg=875229.57, stdev=1466.63, '
                                             'samples=14\n'
                                             '   iops        : min=218318, '
                                             'max=219420, avg=218807.29, '
                                             'stdev=366.41, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=0.06%, '
                                             '20=80.18%, 50=19.53%, 100=0.18%, '
                                             '250=0.04%\n'
                                             '  cpu          : usr=77.81%, '
                                             'sys=22.16%, ctx=22, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.4%, '
                                             '4=99.5%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1531121,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=854MiB/s (896MB/s), '
                                             '854MiB/s-854MiB/s '
                                             '(896MB/s-896MB/s), io=5981MiB '
                                             '(6271MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.output'),
                                   'state': {'begin': 1666727215.3379507,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7949707508087158,
                                             'end': 1666727216.1329215,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_276.state')},
                       'cmd_277': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.output'),
                                   'state': {'begin': 1666727218.1851926,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7935588359832764,
                                             'end': 1666727218.9787514,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_277.state')},
                       'cmd_278': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.output'),
                                   'state': {'begin': 1666727218.981152,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.622839212417603,
                                             'end': 1666727230.6039913,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_278.state')},
                       'cmd_279': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734319,\n'
                                             '  "timestamp_ms" : '
                                             '1666734319976,\n'
                                             '  "time" : "Tue Oct 25 23:45:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6364598272,\n'
                                             '        "io_kbytes" : 6215428,\n'
                                             '        "bw_bytes" : 909098453,\n'
                                             '        "bw" : 887791,\n'
                                             '        "iops" : 221947.293244,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1553853,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1513,\n'
                                             '          "max" : 25969,\n'
                                             '          "mean" : 1631.607934,\n'
                                             '          "stddev" : '
                                             '243.107289,\n'
                                             '          "N" : 1553854\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 261,\n'
                                             '          "max" : 131006,\n'
                                             '          "mean" : '
                                             '16099.061237,\n'
                                             '          "stddev" : '
                                             '5450.853875,\n'
                                             '          "N" : 1553853,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 13504,\n'
                                             '            "5.000000" : 14400,\n'
                                             '            "10.000000" : '
                                             '14528,\n'
                                             '            "20.000000" : '
                                             '14528,\n'
                                             '            "30.000000" : '
                                             '14656,\n'
                                             '            "40.000000" : '
                                             '14656,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '14784,\n'
                                             '            "80.000000" : '
                                             '14912,\n'
                                             '            "90.000000" : '
                                             '16512,\n'
                                             '            "95.000000" : '
                                             '27008,\n'
                                             '            "99.000000" : '
                                             '36096,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '88576,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9708,\n'
                                             '          "max" : 132579,\n'
                                             '          "mean" : '
                                             '17730.669197,\n'
                                             '          "stddev" : '
                                             '5450.382413,\n'
                                             '          "N" : 1553853\n'
                                             '        },\n'
                                             '        "bw_min" : 886560,\n'
                                             '        "bw_max" : 889792,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '888029.857143,\n'
                                             '        "bw_dev" : 1001.853195,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221640,\n'
                                             '        "iops_max" : 222448,\n'
                                             '        "iops_mean" : '
                                             '222007.428571,\n'
                                             '        "iops_stddev" : '
                                             '250.424168,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 69.928571,\n'
                                             '      "sys_cpu" : 30.042857,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.372300,\n'
                                             '        "2" : 0.969268,\n'
                                             '        "4" : 98.658432,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.058757,\n'
                                             '        "20" : 91.962818,\n'
                                             '        "50" : 7.797906,\n'
                                             '        "100" : 0.153618,\n'
                                             '        "250" : 0.024327,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=88418: Tue '
                                             'Oct 25 23:45:19 2022\n'
                                             '  read: IOPS=222k, BW=867MiB/s '
                                             '(909MB/s)(6070MiB/7001msec)\n'
                                             '    slat (nsec): min=1513, '
                                             'max=25969, avg=1631.61, '
                                             'stdev=243.11\n'
                                             '    clat (nsec): min=261, '
                                             'max=131006, avg=16099.06, '
                                             'stdev=5450.85\n'
                                             '     lat (usec): min=9, max=132, '
                                             'avg=17.73, stdev= 5.45\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   14],  '
                                             '5.00th=[   15], 10.00th=[   15], '
                                             '20.00th=[   15],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   15], '
                                             '80.00th=[   15], 90.00th=[   '
                                             '17], 95.00th=[   28],\n'
                                             '     | 99.00th=[   37], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '80], 99.95th=[   89],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=886560, '
                                             'max=889792, per=100.00%, '
                                             'avg=888029.86, stdev=1001.85, '
                                             'samples=14\n'
                                             '   iops        : min=221640, '
                                             'max=222448, avg=222007.43, '
                                             'stdev=250.42, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 10=0.06%, '
                                             '20=91.96%, 50=7.80%, 100=0.15%, '
                                             '250=0.02%\n'
                                             '  cpu          : usr=69.93%, '
                                             'sys=30.04%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.4%, 2=1.0%, '
                                             '4=98.7%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1553853,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=867MiB/s (909MB/s), '
                                             '867MiB/s-867MiB/s '
                                             '(909MB/s-909MB/s), io=6070MiB '
                                             '(6365MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.output'),
                                   'state': {'begin': 1666727230.6067412,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.794581413269043,
                                             'end': 1666727231.4013226,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_279.state')},
                       'cmd_28': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.output'),
                                  'state': {'begin': 1666726028.4532728,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7852113246917725,
                                            'end': 1666726029.2384841,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_28.state')},
                       'cmd_280': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.output'),
                                   'state': {'begin': 1666727233.4648118,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7895946502685547,
                                             'end': 1666727234.2544065,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_280.state')},
                       'cmd_281': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.output'),
                                   'state': {'begin': 1666727234.256411,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.527220249176025,
                                             'end': 1666727245.7836313,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_281.state')},
                       'cmd_282': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734335,\n'
                                             '  "timestamp_ms" : '
                                             '1666734335155,\n'
                                             '  "time" : "Tue Oct 25 23:45:35 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6271553536,\n'
                                             '        "io_kbytes" : 6124564,\n'
                                             '        "bw_bytes" : 895808246,\n'
                                             '        "bw" : 874812,\n'
                                             '        "iops" : 218702.756749,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1531138,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1552,\n'
                                             '          "max" : 172423,\n'
                                             '          "mean" : 2025.800648,\n'
                                             '          "stddev" : '
                                             '791.208125,\n'
                                             '          "N" : 1531138\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 183304,\n'
                                             '          "mean" : '
                                             '15858.742410,\n'
                                             '          "stddev" : '
                                             '5803.263051,\n'
                                             '          "N" : 1531141,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10176,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '13760,\n'
                                             '            "30.000000" : '
                                             '14016,\n'
                                             '            "40.000000" : '
                                             '14272,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '15168,\n'
                                             '            "80.000000" : '
                                             '16192,\n'
                                             '            "90.000000" : '
                                             '17792,\n'
                                             '            "95.000000" : '
                                             '27520,\n'
                                             '            "99.000000" : '
                                             '38656,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '79360,\n'
                                             '            "99.950000" : '
                                             '90624,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9658,\n'
                                             '          "max" : 186690,\n'
                                             '          "mean" : '
                                             '17884.542243,\n'
                                             '          "stddev" : '
                                             '5775.342587,\n'
                                             '          "N" : 1531141\n'
                                             '        },\n'
                                             '        "bw_min" : 872848,\n'
                                             '        "bw_max" : 876888,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '875033.785714,\n'
                                             '        "bw_dev" : 1002.540326,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 218212,\n'
                                             '        "iops_max" : 219222,\n'
                                             '        "iops_mean" : '
                                             '218758.428571,\n'
                                             '        "iops_stddev" : '
                                             '250.748024,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 42.671429,\n'
                                             '      "sys_cpu" : 57.300000,\n'
                                             '      "ctx" : 12,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.273522,\n'
                                             '        "2" : 6.079334,\n'
                                             '        "4" : 93.647143,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.020834,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.145970,\n'
                                             '        "20" : 92.134086,\n'
                                             '        "50" : 7.513366,\n'
                                             '        "100" : 0.155897,\n'
                                             '        "250" : 0.024753,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88468: Tue '
                                             'Oct 25 23:45:35 2022\n'
                                             '  read: IOPS=219k, BW=854MiB/s '
                                             '(896MB/s)(5981MiB/7001msec)\n'
                                             '    slat (nsec): min=1552, '
                                             'max=172423, avg=2025.80, '
                                             'stdev=791.21\n'
                                             '    clat (nsec): min=651, '
                                             'max=183304, avg=15858.74, '
                                             'stdev=5803.26\n'
                                             '     lat (usec): min=9, max=186, '
                                             'avg=17.88, stdev= 5.78\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   11],  '
                                             '5.00th=[   11], 10.00th=[   13], '
                                             '20.00th=[   14],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   16], '
                                             '80.00th=[   17], 90.00th=[   '
                                             '18], 95.00th=[   28],\n'
                                             '     | 99.00th=[   39], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '80], 99.95th=[   91],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=872848, '
                                             'max=876888, per=100.00%, '
                                             'avg=875033.79, stdev=1002.54, '
                                             'samples=14\n'
                                             '   iops        : min=218212, '
                                             'max=219222, avg=218758.43, '
                                             'stdev=250.75, samples=14\n'
                                             '  lat (nsec)   : 750=0.02%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.15%, 20=92.13%, 50=7.51%, '
                                             '100=0.16%\n'
                                             '  lat (usec)   : 250=0.02%\n'
                                             '  cpu          : usr=42.67%, '
                                             'sys=57.30%, ctx=12, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.3%, 2=6.1%, '
                                             '4=93.6%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1531138,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=854MiB/s (896MB/s), '
                                             '854MiB/s-854MiB/s '
                                             '(896MB/s-896MB/s), io=5981MiB '
                                             '(6272MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.output'),
                                   'state': {'begin': 1666727245.7862487,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.789813756942749,
                                             'end': 1666727246.5760624,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_282.state')},
                       'cmd_283': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.output'),
                                   'state': {'begin': 1666727248.5907779,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7839603424072266,
                                             'end': 1666727249.3747382,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_283.state')},
                       'cmd_284': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.output'),
                                   'state': {'begin': 1666727249.3769617,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.562498569488525,
                                             'end': 1666727260.9394603,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_284.state')},
                       'cmd_285': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734350,\n'
                                             '  "timestamp_ms" : '
                                             '1666734350304,\n'
                                             '  "time" : "Tue Oct 25 23:45:50 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6267457536,\n'
                                             '        "io_kbytes" : 6120564,\n'
                                             '        "bw_bytes" : 895223187,\n'
                                             '        "bw" : 874241,\n'
                                             '        "iops" : 218559.777175,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1530137,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1532,\n'
                                             '          "max" : 20007,\n'
                                             '          "mean" : 2014.375794,\n'
                                             '          "stddev" : '
                                             '757.866366,\n'
                                             '          "N" : 1530138\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 641,\n'
                                             '          "max" : 144321,\n'
                                             '          "mean" : '
                                             '15884.766559,\n'
                                             '          "stddev" : '
                                             '5830.698008,\n'
                                             '          "N" : 1530137,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10176,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '13760,\n'
                                             '            "30.000000" : '
                                             '14016,\n'
                                             '            "40.000000" : '
                                             '14272,\n'
                                             '            "50.000000" : '
                                             '14528,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '15168,\n'
                                             '            "80.000000" : '
                                             '16192,\n'
                                             '            "90.000000" : '
                                             '18048,\n'
                                             '            "95.000000" : '
                                             '27520,\n'
                                             '            "99.000000" : '
                                             '38656,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '80384,\n'
                                             '            "99.950000" : '
                                             '90624,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9969,\n'
                                             '          "max" : 145914,\n'
                                             '          "mean" : '
                                             '17899.142622,\n'
                                             '          "stddev" : '
                                             '5795.903965,\n'
                                             '          "N" : 1530137\n'
                                             '        },\n'
                                             '        "bw_min" : 871464,\n'
                                             '        "bw_max" : 876000,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '874451.071429,\n'
                                             '        "bw_dev" : 1455.933772,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 217866,\n'
                                             '        "iops_max" : 219000,\n'
                                             '        "iops_mean" : '
                                             '218612.857143,\n'
                                             '        "iops_stddev" : '
                                             '364.172161,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.014286,\n'
                                             '      "sys_cpu" : 58.957143,\n'
                                             '      "ctx" : 6,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.280106,\n'
                                             '        "2" : 6.138274,\n'
                                             '        "4" : 93.581620,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.018953,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.173122,\n'
                                             '        "20" : 92.036465,\n'
                                             '        "50" : 7.573636,\n'
                                             '        "100" : 0.166521,\n'
                                             '        "250" : 0.026403,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88518: Tue '
                                             'Oct 25 23:45:50 2022\n'
                                             '  read: IOPS=219k, BW=854MiB/s '
                                             '(895MB/s)(5977MiB/7001msec)\n'
                                             '    slat (nsec): min=1532, '
                                             'max=20007, avg=2014.38, '
                                             'stdev=757.87\n'
                                             '    clat (nsec): min=641, '
                                             'max=144321, avg=15884.77, '
                                             'stdev=5830.70\n'
                                             '     lat (usec): min=9, max=145, '
                                             'avg=17.90, stdev= 5.80\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   11],  '
                                             '5.00th=[   11], 10.00th=[   13], '
                                             '20.00th=[   14],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   16], '
                                             '80.00th=[   17], 90.00th=[   '
                                             '19], 95.00th=[   28],\n'
                                             '     | 99.00th=[   39], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '81], 99.95th=[   91],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=871464, '
                                             'max=876000, per=100.00%, '
                                             'avg=874451.07, stdev=1455.93, '
                                             'samples=14\n'
                                             '   iops        : min=217866, '
                                             'max=219000, avg=218612.86, '
                                             'stdev=364.17, samples=14\n'
                                             '  lat (nsec)   : 750=0.02%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.17%, 20=92.04%, 50=7.57%, '
                                             '100=0.17%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=41.01%, '
                                             'sys=58.96%, ctx=6, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.3%, 2=6.1%, '
                                             '4=93.6%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1530137,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=854MiB/s (895MB/s), '
                                             '854MiB/s-854MiB/s '
                                             '(895MB/s-895MB/s), io=5977MiB '
                                             '(6267MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.output'),
                                   'state': {'begin': 1666727260.9420917,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7870001792907715,
                                             'end': 1666727261.729092,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_285.state')},
                       'cmd_286': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.output'),
                                   'state': {'begin': 1666727263.7560027,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7789556980133057,
                                             'end': 1666727264.5349584,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_286.state')},
                       'cmd_287': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.output'),
                                   'state': {'begin': 1666727264.5373862,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.526393175125122,
                                             'end': 1666727276.0637794,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_287.state')},
                       'cmd_288': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734365,\n'
                                             '  "timestamp_ms" : '
                                             '1666734365443,\n'
                                             '  "time" : "Tue Oct 25 23:46:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6263152640,\n'
                                             '        "io_kbytes" : 6116360,\n'
                                             '        "bw_bytes" : 894608290,\n'
                                             '        "bw" : 873640,\n'
                                             '        "iops" : 218409.798600,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1529087,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1552,\n'
                                             '          "max" : 1296060,\n'
                                             '          "mean" : 2038.297024,\n'
                                             '          "stddev" : '
                                             '1293.969014,\n'
                                             '          "N" : 1529087\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 652,\n'
                                             '          "max" : 1309395,\n'
                                             '          "mean" : '
                                             '15867.342854,\n'
                                             '          "stddev" : '
                                             '6045.816309,\n'
                                             '          "N" : 1529090,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 10304,\n'
                                             '            "5.000000" : 10560,\n'
                                             '            "10.000000" : '
                                             '12992,\n'
                                             '            "20.000000" : '
                                             '13760,\n'
                                             '            "30.000000" : '
                                             '14016,\n'
                                             '            "40.000000" : '
                                             '14400,\n'
                                             '            "50.000000" : '
                                             '14656,\n'
                                             '            "60.000000" : '
                                             '14784,\n'
                                             '            "70.000000" : '
                                             '15168,\n'
                                             '            "80.000000" : '
                                             '16192,\n'
                                             '            "90.000000" : '
                                             '18048,\n'
                                             '            "95.000000" : '
                                             '27520,\n'
                                             '            "99.000000" : '
                                             '38656,\n'
                                             '            "99.500000" : '
                                             '42752,\n'
                                             '            "99.900000" : '
                                             '78336,\n'
                                             '            "99.950000" : '
                                             '90624,\n'
                                             '            "99.990000" : '
                                             '103936\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 9738,\n'
                                             '          "max" : 1311068,\n'
                                             '          "mean" : '
                                             '17905.639056,\n'
                                             '          "stddev" : '
                                             '6101.138943,\n'
                                             '          "N" : 1529090\n'
                                             '        },\n'
                                             '        "bw_min" : 871208,\n'
                                             '        "bw_max" : 876849,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '873847.500000,\n'
                                             '        "bw_dev" : 1583.292322,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 217802,\n'
                                             '        "iops_max" : 219212,\n'
                                             '        "iops_mean" : '
                                             '218461.857143,\n'
                                             '        "iops_stddev" : '
                                             '395.786628,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.414286,\n'
                                             '      "sys_cpu" : 58.557143,\n'
                                             '      "ctx" : 45,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.279579,\n'
                                             '        "2" : 6.606622,\n'
                                             '        "4" : 93.113799,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.020470,\n'
                                             '        "1000" : 0.010000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.132170,\n'
                                             '        "20" : 92.125039,\n'
                                             '        "50" : 7.538289,\n'
                                             '        "100" : 0.154994,\n'
                                             '        "250" : 0.023413,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88567: Tue '
                                             'Oct 25 23:46:05 2022\n'
                                             '  read: IOPS=218k, BW=853MiB/s '
                                             '(895MB/s)(5973MiB/7001msec)\n'
                                             '    slat (nsec): min=1552, '
                                             'max=1296.1k, avg=2038.30, '
                                             'stdev=1293.97\n'
                                             '    clat (nsec): min=652, '
                                             'max=1309.4k, avg=15867.34, '
                                             'stdev=6045.82\n'
                                             '     lat (usec): min=9, '
                                             'max=1311, avg=17.91, stdev= '
                                             '6.10\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   11],  '
                                             '5.00th=[   11], 10.00th=[   13], '
                                             '20.00th=[   14],\n'
                                             '     | 30.00th=[   15], '
                                             '40.00th=[   15], 50.00th=[   '
                                             '15], 60.00th=[   15],\n'
                                             '     | 70.00th=[   16], '
                                             '80.00th=[   17], 90.00th=[   '
                                             '19], 95.00th=[   28],\n'
                                             '     | 99.00th=[   39], '
                                             '99.50th=[   43], 99.90th=[   '
                                             '79], 99.95th=[   91],\n'
                                             '     | 99.99th=[  104]\n'
                                             '   bw (  KiB/s): min=871208, '
                                             'max=876849, per=100.00%, '
                                             'avg=873847.50, stdev=1583.29, '
                                             'samples=14\n'
                                             '   iops        : min=217802, '
                                             'max=219212, avg=218461.86, '
                                             'stdev=395.79, samples=14\n'
                                             '  lat (nsec)   : 750=0.02%, '
                                             '1000=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.13%, 20=92.13%, 50=7.54%, '
                                             '100=0.15%\n'
                                             '  lat (usec)   : 250=0.02%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=41.41%, '
                                             'sys=58.56%, ctx=45, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.3%, 2=6.6%, '
                                             '4=93.1%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1529087,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=853MiB/s (895MB/s), '
                                             '853MiB/s-853MiB/s '
                                             '(895MB/s-895MB/s), io=5973MiB '
                                             '(6263MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.output'),
                                   'state': {'begin': 1666727276.0664513,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7896168231964111,
                                             'end': 1666727276.8560681,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_288.state')},
                       'cmd_289': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.output'),
                                   'state': {'begin': 1666727277.8519075,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7837321758270264,
                                             'end': 1666727278.6356397,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_289.state')},
                       'cmd_29': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.output'),
                                  'state': {'begin': 1666726029.2404494,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-reference_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.047251462936401,
                                            'end': 1666726040.287701,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_29.state')},
                       'cmd_290': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.output'),
                                   'state': {'begin': 1666727278.637711,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.082463502883911,
                                             'end': 1666727289.7201746,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_290.state')},
                       'cmd_291': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734379,\n'
                                             '  "timestamp_ms" : '
                                             '1666734379201,\n'
                                             '  "time" : "Tue Oct 25 23:46:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5072072704,\n'
                                             '        "io_kbytes" : 4953196,\n'
                                             '        "bw_bytes" : 724478317,\n'
                                             '        "bw" : 707498,\n'
                                             '        "iops" : 176874.160834,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1238296,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1392,\n'
                                             '          "max" : 24916,\n'
                                             '          "mean" : 2289.736705,\n'
                                             '          "stddev" : '
                                             '1489.650647,\n'
                                             '          "N" : 1238296\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2695,\n'
                                             '          "max" : 133280,\n'
                                             '          "mean" : '
                                             '20010.147292,\n'
                                             '          "stddev" : '
                                             '5825.442065,\n'
                                             '          "N" : 1238299,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14912,\n'
                                             '            "5.000000" : 15680,\n'
                                             '            "10.000000" : '
                                             '15808,\n'
                                             '            "20.000000" : '
                                             '16064,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '19328,\n'
                                             '            "60.000000" : '
                                             '19584,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22144,\n'
                                             '            "95.000000" : '
                                             '31360,\n'
                                             '            "99.000000" : '
                                             '43776,\n'
                                             '            "99.500000" : '
                                             '46848,\n'
                                             '            "99.900000" : '
                                             '81408,\n'
                                             '            "99.950000" : '
                                             '93696,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10369,\n'
                                             '          "max" : 134742,\n'
                                             '          "mean" : '
                                             '22299.882204,\n'
                                             '          "stddev" : '
                                             '5670.530977,\n'
                                             '          "N" : 1238299\n'
                                             '        },\n'
                                             '        "bw_min" : 703320,\n'
                                             '        "bw_max" : 710557,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '707687.785714,\n'
                                             '        "bw_dev" : 1699.738812,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 175830,\n'
                                             '        "iops_max" : 177639,\n'
                                             '        "iops_mean" : '
                                             '176921.928571,\n'
                                             '        "iops_stddev" : '
                                             '424.902245,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.414286,\n'
                                             '      "sys_cpu" : 50.042857,\n'
                                             '      "ctx" : 319314,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.027215,\n'
                                             '        "20" : 65.634549,\n'
                                             '        "50" : 34.085227,\n'
                                             '        "100" : 0.224744,\n'
                                             '        "250" : 0.028345,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88606: Tue '
                                             'Oct 25 23:46:19 2022\n'
                                             '  read: IOPS=177k, BW=691MiB/s '
                                             '(724MB/s)(4837MiB/7001msec)\n'
                                             '    slat (nsec): min=1392, '
                                             'max=24916, avg=2289.74, '
                                             'stdev=1489.65\n'
                                             '    clat (usec): min=2, max=133, '
                                             'avg=20.01, stdev= 5.83\n'
                                             '     lat (usec): min=10, '
                                             'max=134, avg=22.30, stdev= 5.67\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   16], 10.00th=[   16], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '20], 60.00th=[   20],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '23], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   47], 99.90th=[   '
                                             '82], 99.95th=[   94],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=703320, '
                                             'max=710557, per=100.00%, '
                                             'avg=707687.79, stdev=1699.74, '
                                             'samples=14\n'
                                             '   iops        : min=175830, '
                                             'max=177639, avg=176921.93, '
                                             'stdev=424.90, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=65.63%, 50=34.09%, '
                                             '100=0.22%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=15.41%, '
                                             'sys=50.04%, ctx=319314, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1238296,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=691MiB/s (724MB/s), '
                                             '691MiB/s-691MiB/s '
                                             '(724MB/s-724MB/s), io=4837MiB '
                                             '(5072MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.output'),
                                   'state': {'begin': 1666727289.723327,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.789381742477417,
                                             'end': 1666727290.5127087,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_291.state')},
                       'cmd_292': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.output'),
                                   'state': {'begin': 1666727291.4978764,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7889368534088135,
                                             'end': 1666727292.2868133,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_292.state')},
                       'cmd_293': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.output'),
                                   'state': {'begin': 1666727292.289012,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.090299606323242,
                                             'end': 1666727303.3793116,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_293.state')},
                       'cmd_294': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734392,\n'
                                             '  "timestamp_ms" : '
                                             '1666734392857,\n'
                                             '  "time" : "Tue Oct 25 23:46:32 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5079027712,\n'
                                             '        "io_kbytes" : 4959988,\n'
                                             '        "bw_bytes" : 725471748,\n'
                                             '        "bw" : 708468,\n'
                                             '        "iops" : 177116.554778,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1239993,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1432,\n'
                                             '          "max" : 25278,\n'
                                             '          "mean" : 2296.353298,\n'
                                             '          "stddev" : '
                                             '1460.013791,\n'
                                             '          "N" : 1239994\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5000,\n'
                                             '          "max" : 315411,\n'
                                             '          "mean" : '
                                             '19955.888521,\n'
                                             '          "stddev" : '
                                             '5758.486187,\n'
                                             '          "N" : 1239996,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14656,\n'
                                             '            "5.000000" : 15808,\n'
                                             '            "10.000000" : '
                                             '15808,\n'
                                             '            "20.000000" : '
                                             '16064,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '19328,\n'
                                             '            "60.000000" : '
                                             '19584,\n'
                                             '            "70.000000" : '
                                             '21376,\n'
                                             '            "80.000000" : '
                                             '21632,\n'
                                             '            "90.000000" : '
                                             '21888,\n'
                                             '            "95.000000" : '
                                             '31360,\n'
                                             '            "99.000000" : '
                                             '43264,\n'
                                             '            "99.500000" : '
                                             '46336,\n'
                                             '            "99.900000" : '
                                             '76288,\n'
                                             '            "99.950000" : '
                                             '92672,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10500,\n'
                                             '          "max" : 316904,\n'
                                             '          "mean" : '
                                             '22252.243399,\n'
                                             '          "stddev" : '
                                             '5586.706939,\n'
                                             '          "N" : 1239996\n'
                                             '        },\n'
                                             '        "bw_min" : 704384,\n'
                                             '        "bw_max" : 710608,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '708637.857143,\n'
                                             '        "bw_dev" : 1613.810486,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 176096,\n'
                                             '        "iops_max" : 177652,\n'
                                             '        "iops_mean" : '
                                             '177159.285714,\n'
                                             '        "iops_stddev" : '
                                             '403.671311,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.328571,\n'
                                             '      "sys_cpu" : 49.528571,\n'
                                             '      "ctx" : 320288,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.015161,\n'
                                             '        "20" : 66.261019,\n'
                                             '        "50" : 33.495592,\n'
                                             '        "100" : 0.202824,\n'
                                             '        "250" : 0.025323,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88647: Tue '
                                             'Oct 25 23:46:32 2022\n'
                                             '  read: IOPS=177k, BW=692MiB/s '
                                             '(725MB/s)(4844MiB/7001msec)\n'
                                             '    slat (nsec): min=1432, '
                                             'max=25278, avg=2296.35, '
                                             'stdev=1460.01\n'
                                             '    clat (usec): min=5, max=315, '
                                             'avg=19.96, stdev= 5.76\n'
                                             '     lat (usec): min=10, '
                                             'max=316, avg=22.25, stdev= 5.59\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   16], 10.00th=[   16], '
                                             '20.00th=[   17],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '20], 60.00th=[   20],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '22], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   47], 99.90th=[   '
                                             '77], 99.95th=[   93],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=704384, '
                                             'max=710608, per=100.00%, '
                                             'avg=708637.86, stdev=1613.81, '
                                             'samples=14\n'
                                             '   iops        : min=176096, '
                                             'max=177652, avg=177159.29, '
                                             'stdev=403.67, samples=14\n'
                                             '  lat (usec)   : 10=0.02%, '
                                             '20=66.26%, 50=33.50%, 100=0.20%, '
                                             '250=0.03%\n'
                                             '  lat (usec)   : 500=0.01%\n'
                                             '  cpu          : usr=16.33%, '
                                             'sys=49.53%, ctx=320288, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1239993,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=692MiB/s (725MB/s), '
                                             '692MiB/s-692MiB/s '
                                             '(725MB/s-725MB/s), io=4844MiB '
                                             '(5079MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.output'),
                                   'state': {'begin': 1666727303.386083,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8056910037994385,
                                             'end': 1666727304.191774,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_294.state')},
                       'cmd_295': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_4_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.output'),
                                   'state': {'begin': 1666727305.191382,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7836463451385498,
                                             'end': 1666727305.9750283,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_295.state')},
                       'cmd_296': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.output'),
                                   'state': {'begin': 1666727305.977193,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="4" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_4_libaio-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.102250337600708,
                                             'end': 1666727317.0794435,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_296.state')},
                       'cmd_297': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=4\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734406,\n'
                                             '  "timestamp_ms" : '
                                             '1666734406559,\n'
                                             '  "time" : "Tue Oct 25 23:46:46 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "4",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '5081395200,\n'
                                             '        "io_kbytes" : 4962300,\n'
                                             '        "bw_bytes" : 725809912,\n'
                                             '        "bw" : 708798,\n'
                                             '        "iops" : 177199.114412,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1240571,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1402,\n'
                                             '          "max" : 30307,\n'
                                             '          "mean" : 2212.269113,\n'
                                             '          "stddev" : '
                                             '1350.794632,\n'
                                             '          "N" : 1240572\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2756,\n'
                                             '          "max" : 142718,\n'
                                             '          "mean" : '
                                             '20076.305615,\n'
                                             '          "stddev" : '
                                             '5924.159167,\n'
                                             '          "N" : 1240574,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 14144,\n'
                                             '            "5.000000" : 14784,\n'
                                             '            "10.000000" : '
                                             '15040,\n'
                                             '            "20.000000" : '
                                             '15936,\n'
                                             '            "30.000000" : '
                                             '17280,\n'
                                             '            "40.000000" : '
                                             '17536,\n'
                                             '            "50.000000" : '
                                             '19328,\n'
                                             '            "60.000000" : '
                                             '21376,\n'
                                             '            "70.000000" : '
                                             '21632,\n'
                                             '            "80.000000" : '
                                             '21888,\n'
                                             '            "90.000000" : '
                                             '22144,\n'
                                             '            "95.000000" : '
                                             '31360,\n'
                                             '            "99.000000" : '
                                             '43776,\n'
                                             '            "99.500000" : '
                                             '46336,\n'
                                             '            "99.900000" : '
                                             '80384,\n'
                                             '            "99.950000" : '
                                             '94720,\n'
                                             '            "99.990000" : '
                                             '105984\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 10880,\n'
                                             '          "max" : 144201,\n'
                                             '          "mean" : '
                                             '22288.576315,\n'
                                             '          "stddev" : '
                                             '6010.338472,\n'
                                             '          "N" : 1240574\n'
                                             '        },\n'
                                             '        "bw_min" : 700656,\n'
                                             '        "bw_max" : 710264,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '708977.357143,\n'
                                             '        "bw_dev" : 2422.535722,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 175164,\n'
                                             '        "iops_max" : 177566,\n'
                                             '        "iops_mean" : '
                                             '177244.428571,\n'
                                             '        "iops_stddev" : '
                                             '605.618910,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.228571,\n'
                                             '      "sys_cpu" : 50.371429,\n'
                                             '      "ctx" : 318778,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.026842,\n'
                                             '        "20" : 57.241706,\n'
                                             '        "50" : 42.487371,\n'
                                             '        "100" : 0.212402,\n'
                                             '        "250" : 0.031760,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 4,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88686: Tue '
                                             'Oct 25 23:46:46 2022\n'
                                             '  read: IOPS=177k, BW=692MiB/s '
                                             '(726MB/s)(4846MiB/7001msec)\n'
                                             '    slat (nsec): min=1402, '
                                             'max=30307, avg=2212.27, '
                                             'stdev=1350.79\n'
                                             '    clat (usec): min=2, max=142, '
                                             'avg=20.08, stdev= 5.92\n'
                                             '     lat (usec): min=10, '
                                             'max=144, avg=22.29, stdev= 6.01\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   15],  '
                                             '5.00th=[   15], 10.00th=[   16], '
                                             '20.00th=[   16],\n'
                                             '     | 30.00th=[   18], '
                                             '40.00th=[   18], 50.00th=[   '
                                             '20], 60.00th=[   22],\n'
                                             '     | 70.00th=[   22], '
                                             '80.00th=[   22], 90.00th=[   '
                                             '23], 95.00th=[   32],\n'
                                             '     | 99.00th=[   44], '
                                             '99.50th=[   47], 99.90th=[   '
                                             '81], 99.95th=[   95],\n'
                                             '     | 99.99th=[  106]\n'
                                             '   bw (  KiB/s): min=700656, '
                                             'max=710264, per=100.00%, '
                                             'avg=708977.36, stdev=2422.54, '
                                             'samples=14\n'
                                             '   iops        : min=175164, '
                                             'max=177566, avg=177244.43, '
                                             'stdev=605.62, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.03%, 20=57.24%, 50=42.49%, '
                                             '100=0.21%\n'
                                             '  lat (usec)   : 250=0.03%\n'
                                             '  cpu          : usr=14.23%, '
                                             'sys=50.37%, ctx=318778, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1240571,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=4\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=692MiB/s (726MB/s), '
                                             '692MiB/s-692MiB/s '
                                             '(726MB/s-726MB/s), io=4846MiB '
                                             '(5081MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.output'),
                                   'state': {'begin': 1666727317.082374,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_4_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7876636981964111,
                                             'end': 1666727317.8700378,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_297.state')},
                       'cmd_298': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.output'),
                                   'state': {'begin': 1666727318.8601072,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.793532133102417,
                                             'end': 1666727319.6536393,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_298.state')},
                       'cmd_299': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.output'),
                                   'state': {'begin': 1666727319.6556458,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.054964542388916,
                                             'end': 1666727330.7106104,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_299.state')},
                       'cmd_30': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733129,\n'
                                            '  "timestamp_ms" : '
                                            '1666733129789,\n'
                                            '  "time" : "Tue Oct 25 23:25:29 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1804812288,\n'
                                            '        "io_kbytes" : 1762512,\n'
                                            '        "bw_bytes" : 257793499,\n'
                                            '        "bw" : 251751,\n'
                                            '        "iops" : 62937.723182,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 440627,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2425,\n'
                                            '          "max" : 42410,\n'
                                            '          "mean" : 2632.123117,\n'
                                            '          "stddev" : 277.348007,\n'
                                            '          "N" : 440628\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 210,\n'
                                            '          "max" : 185869,\n'
                                            '          "mean" : 12588.789237,\n'
                                            '          "stddev" : '
                                            '2063.603963,\n'
                                            '          "N" : 440627,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 12096,\n'
                                            '            "5.000000" : 12224,\n'
                                            '            "10.000000" : 12224,\n'
                                            '            "20.000000" : 12224,\n'
                                            '            "30.000000" : 12224,\n'
                                            '            "40.000000" : 12352,\n'
                                            '            "50.000000" : 12352,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22400,\n'
                                            '            "99.500000" : 28032,\n'
                                            '            "99.900000" : 39168,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 66048\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12183,\n'
                                            '          "max" : 188574,\n'
                                            '          "mean" : 15220.912393,\n'
                                            '          "stddev" : '
                                            '2084.834946,\n'
                                            '          "N" : 440627\n'
                                            '        },\n'
                                            '        "bw_min" : 250536,\n'
                                            '        "bw_max" : 252744,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '251847.142857,\n'
                                            '        "bw_dev" : 664.546445,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 62634,\n'
                                            '        "iops_max" : 63186,\n'
                                            '        "iops_mean" : '
                                            '62961.714286,\n'
                                            '        "iops_stddev" : '
                                            '166.188229,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.728571,\n'
                                            '      "sys_cpu" : 38.085714,\n'
                                            '      "ctx" : 440609,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.589737,\n'
                                            '        "50" : 1.345809,\n'
                                            '        "100" : 0.053787,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 630140,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 6063,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 6063,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=84772: Tue '
                                            'Oct 25 23:25:29 2022\n'
                                            '  read: IOPS=62.9k, BW=246MiB/s '
                                            '(258MB/s)(1721MiB/7001msec)\n'
                                            '    slat (nsec): min=2425, '
                                            'max=42410, avg=2632.12, '
                                            'stdev=277.35\n'
                                            '    clat (nsec): min=210, '
                                            'max=185869, avg=12588.79, '
                                            'stdev=2063.60\n'
                                            '     lat (usec): min=12, max=188, '
                                            'avg=15.22, stdev= 2.08\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[12096],  '
                                            '5.00th=[12224], 10.00th=[12224], '
                                            '20.00th=[12224],\n'
                                            '     | 30.00th=[12224], '
                                            '40.00th=[12352], 50.00th=[12352], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12480], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22400], '
                                            '99.50th=[28032], 99.90th=[39168], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[66048]\n'
                                            '   bw (  KiB/s): min=250536, '
                                            'max=252744, per=100.00%, '
                                            'avg=251847.14, stdev=664.55, '
                                            'samples=14\n'
                                            '   iops        : min=62634, '
                                            'max=63186, avg=62961.71, '
                                            'stdev=166.19, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.59%, 50=1.35%, '
                                            '100=0.05%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=9.73%, '
                                            'sys=38.09%, ctx=440609, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=440627,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=246MiB/s (258MB/s), '
                                            '246MiB/s-246MiB/s '
                                            '(258MB/s-258MB/s), io=1721MiB '
                                            '(1805MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=630140/0, '
                                            'merge=0/0, ticks=6063/0, '
                                            'in_queue=6063, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.output'),
                                  'state': {'begin': 1666726040.2903283,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7855043411254883,
                                            'end': 1666726041.0758326,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_30.state')},
                       'cmd_300': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734420,\n'
                                             '  "timestamp_ms" : '
                                             '1666734420189,\n'
                                             '  "time" : "Tue Oct 25 23:47:00 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6420176896,\n'
                                             '        "io_kbytes" : 6269704,\n'
                                             '        "bw_bytes" : 917037122,\n'
                                             '        "bw" : 895544,\n'
                                             '        "iops" : 223884.873589,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567418,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1082,\n'
                                             '          "max" : 24687,\n'
                                             '          "mean" : 1805.728822,\n'
                                             '          "stddev" : '
                                             '832.327889,\n'
                                             '          "N" : 1567419\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11222,\n'
                                             '          "max" : 215775,\n'
                                             '          "mean" : '
                                             '33639.284363,\n'
                                             '          "stddev" : '
                                             '8658.611942,\n'
                                             '          "N" : 1567425,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24960,\n'
                                             '            "5.000000" : 28800,\n'
                                             '            "10.000000" : '
                                             '29568,\n'
                                             '            "20.000000" : '
                                             '30080,\n'
                                             '            "30.000000" : '
                                             '30080,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '31872,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '127488,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15389,\n'
                                             '          "max" : 218299,\n'
                                             '          "mean" : '
                                             '35445.013616,\n'
                                             '          "stddev" : '
                                             '8573.177387,\n'
                                             '          "N" : 1567425\n'
                                             '        },\n'
                                             '        "bw_min" : 893760,\n'
                                             '        "bw_max" : 898008,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895765.357143,\n'
                                             '        "bw_dev" : 1348.294627,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223440,\n'
                                             '        "iops_max" : 224500,\n'
                                             '        "iops_mean" : '
                                             '223941.285714,\n'
                                             '        "iops_stddev" : '
                                             '336.789920,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.514286,\n'
                                             '      "sys_cpu" : 58.171429,\n'
                                             '      "ctx" : 756092,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.102462,\n'
                                             '        "50" : 91.998178,\n'
                                             '        "100" : 7.644419,\n'
                                             '        "250" : 0.255388,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=88726: Tue Oct 25 23:47:00 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6123MiB/7001msec)\n'
                                             '    slat (nsec): min=1082, '
                                             'max=24687, avg=1805.73, '
                                             'stdev=832.33\n'
                                             '    clat (usec): min=11, '
                                             'max=215, avg=33.64, stdev= 8.66\n'
                                             '     lat (usec): min=15, '
                                             'max=218, avg=35.45, stdev= 8.57\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   30], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   53],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '128], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893760, '
                                             'max=898008, per=100.00%, '
                                             'avg=895765.36, stdev=1348.29, '
                                             'samples=14\n'
                                             '   iops        : min=223440, '
                                             'max=224500, avg=223941.29, '
                                             'stdev=336.79, samples=14\n'
                                             '  lat (usec)   : 20=0.10%, '
                                             '50=92.00%, 100=7.64%, 250=0.26%\n'
                                             '  cpu          : usr=14.51%, '
                                             'sys=58.17%, ctx=756092, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567418,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6123MiB '
                                             '(6420MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.output'),
                                   'state': {'begin': 1666727330.713518,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.781747579574585,
                                             'end': 1666727331.4952655,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_300.state')},
                       'cmd_301': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.output'),
                                   'state': {'begin': 1666727332.520811,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.785956621170044,
                                             'end': 1666727333.3067677,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_301.state')},
                       'cmd_302': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.output'),
                                   'state': {'begin': 1666727333.309005,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.057814359664917,
                                             'end': 1666727344.3668194,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_302.state')},
                       'cmd_303': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734433,\n'
                                             '  "timestamp_ms" : '
                                             '1666734433845,\n'
                                             '  "time" : "Tue Oct 25 23:47:13 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6416601088,\n'
                                             '        "io_kbytes" : 6266212,\n'
                                             '        "bw_bytes" : 916526365,\n'
                                             '        "bw" : 895045,\n'
                                             '        "iops" : 223760.319954,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566546,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1102,\n'
                                             '          "max" : 42740,\n'
                                             '          "mean" : 1833.665654,\n'
                                             '          "stddev" : '
                                             '837.174942,\n'
                                             '          "N" : 1566546\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 4929,\n'
                                             '          "max" : 212439,\n'
                                             '          "mean" : '
                                             '33633.179418,\n'
                                             '          "stddev" : '
                                             '8764.799429,\n'
                                             '          "N" : 1566553,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 28800,\n'
                                             '            "10.000000" : '
                                             '29568,\n'
                                             '            "20.000000" : '
                                             '30080,\n'
                                             '            "30.000000" : '
                                             '30080,\n'
                                             '            "40.000000" : '
                                             '30592,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '31872,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '68096,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15479,\n'
                                             '          "max" : 215695,\n'
                                             '          "mean" : '
                                             '35466.844208,\n'
                                             '          "stddev" : '
                                             '8680.017687,\n'
                                             '          "N" : 1566553\n'
                                             '        },\n'
                                             '        "bw_min" : 891968,\n'
                                             '        "bw_max" : 897104,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895253.928571,\n'
                                             '        "bw_dev" : 1517.357950,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 222992,\n'
                                             '        "iops_max" : 224276,\n'
                                             '        "iops_mean" : '
                                             '223813.428571,\n'
                                             '        "iops_stddev" : '
                                             '379.390141,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 13.785714,\n'
                                             '      "sys_cpu" : 58.928571,\n'
                                             '      "ctx" : 753694,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.106285,\n'
                                             '        "50" : 92.040195,\n'
                                             '        "100" : 7.578456,\n'
                                             '        "250" : 0.275447,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=88764: Tue Oct 25 23:47:13 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6119MiB/7001msec)\n'
                                             '    slat (nsec): min=1102, '
                                             'max=42740, avg=1833.67, '
                                             'stdev=837.17\n'
                                             '    clat (usec): min=4, max=212, '
                                             'avg=33.63, stdev= 8.76\n'
                                             '     lat (usec): min=15, '
                                             'max=215, avg=35.47, stdev= 8.68\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   30], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   53],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   69], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=891968, '
                                             'max=897104, per=100.00%, '
                                             'avg=895253.93, stdev=1517.36, '
                                             'samples=14\n'
                                             '   iops        : min=222992, '
                                             'max=224276, avg=223813.43, '
                                             'stdev=379.39, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.11%, 50=92.04%, 100=7.58%, '
                                             '250=0.28%\n'
                                             '  cpu          : usr=13.79%, '
                                             'sys=58.93%, ctx=753694, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566546,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6119MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.output'),
                                   'state': {'begin': 1666727344.3698432,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.785027027130127,
                                             'end': 1666727345.1548703,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_303.state')},
                       'cmd_304': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.output'),
                                   'state': {'begin': 1666727346.194987,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7897701263427734,
                                             'end': 1666727346.9847572,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_304.state')},
                       'cmd_305': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.output'),
                                   'state': {'begin': 1666727346.9870663,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring_cmd" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.042422533035278,
                                             'end': 1666727358.0294888,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_305.state')},
                       'cmd_306': {'output': 'io_uring_cmd-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, '
                                             'ioengine=io_uring_cmd, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734447,\n'
                                             '  "timestamp_ms" : '
                                             '1666734447509,\n'
                                             '  "time" : "Tue Oct 25 23:47:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring_cmd",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6417027072,\n'
                                             '        "io_kbytes" : 6266628,\n'
                                             '        "bw_bytes" : 916587212,\n'
                                             '        "bw" : 895104,\n'
                                             '        "iops" : 223775.032138,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566649,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1082,\n'
                                             '          "max" : 28684,\n'
                                             '          "mean" : 1801.328699,\n'
                                             '          "stddev" : '
                                             '837.999667,\n'
                                             '          "N" : 1566650\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11131,\n'
                                             '          "max" : 206056,\n'
                                             '          "mean" : '
                                             '33649.558555,\n'
                                             '          "stddev" : '
                                             '8666.871644,\n'
                                             '          "N" : 1566656,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 28800,\n'
                                             '            "10.000000" : '
                                             '29824,\n'
                                             '            "20.000000" : '
                                             '30080,\n'
                                             '            "30.000000" : '
                                             '30336,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '31872,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '128512,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15359,\n'
                                             '          "max" : 208260,\n'
                                             '          "mean" : '
                                             '35450.887665,\n'
                                             '          "stddev" : '
                                             '8587.323029,\n'
                                             '          "N" : 1566656\n'
                                             '        },\n'
                                             '        "bw_min" : 885416,\n'
                                             '        "bw_max" : 897224,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895327.500000,\n'
                                             '        "bw_dev" : 3021.324589,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221354,\n'
                                             '        "iops_max" : 224306,\n'
                                             '        "iops_mean" : '
                                             '223831.857143,\n'
                                             '        "iops_stddev" : '
                                             '755.321415,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 14.185714,\n'
                                             '      "sys_cpu" : 58.757143,\n'
                                             '      "ctx" : 755492,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.092554,\n'
                                             '        "50" : 92.064145,\n'
                                             '        "100" : 7.584341,\n'
                                             '        "250" : 0.259407,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-reference: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=88804: Tue Oct 25 23:47:27 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6120MiB/7001msec)\n'
                                             '    slat (nsec): min=1082, '
                                             'max=28684, avg=1801.33, '
                                             'stdev=838.00\n'
                                             '    clat (usec): min=11, '
                                             'max=206, avg=33.65, stdev= 8.67\n'
                                             '     lat (usec): min=15, '
                                             'max=208, avg=35.45, stdev= 8.59\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   30], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '129], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=885416, '
                                             'max=897224, per=100.00%, '
                                             'avg=895327.50, stdev=3021.32, '
                                             'samples=14\n'
                                             '   iops        : min=221354, '
                                             'max=224306, avg=223831.86, '
                                             'stdev=755.32, samples=14\n'
                                             '  lat (usec)   : 20=0.09%, '
                                             '50=92.06%, 100=7.58%, 250=0.26%\n'
                                             '  cpu          : usr=14.19%, '
                                             'sys=58.76%, ctx=755492, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566649,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6120MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.output'),
                                   'state': {'begin': 1666727358.0324516,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7928454875946045,
                                             'end': 1666727358.825297,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_306.state')},
                       'cmd_307': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.output'),
                                   'state': {'begin': 1666727359.870337,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7896568775177002,
                                             'end': 1666727360.659994,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_307.state')},
                       'cmd_308': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.output'),
                                   'state': {'begin': 1666727360.6621885,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.059838771820068,
                                             'end': 1666727371.7220273,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_308.state')},
                       'cmd_309': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734461,\n'
                                             '  "timestamp_ms" : '
                                             '1666734461202,\n'
                                             '  "time" : "Tue Oct 25 23:47:41 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6421319680,\n'
                                             '        "io_kbytes" : 6270820,\n'
                                             '        "bw_bytes" : 917200354,\n'
                                             '        "bw" : 895703,\n'
                                             '        "iops" : 223924.725039,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567697,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1152,\n'
                                             '          "max" : 175619,\n'
                                             '          "mean" : 1747.089774,\n'
                                             '          "stddev" : '
                                             '791.058747,\n'
                                             '          "N" : 1567698\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2465,\n'
                                             '          "max" : 215894,\n'
                                             '          "mean" : '
                                             '33673.439684,\n'
                                             '          "stddev" : '
                                             '8679.324358,\n'
                                             '          "N" : 1567704,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24960,\n'
                                             '            "5.000000" : 28800,\n'
                                             '            "10.000000" : '
                                             '29056,\n'
                                             '            "20.000000" : '
                                             '29824,\n'
                                             '            "30.000000" : '
                                             '30336,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '31872,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '158720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15198,\n'
                                             '          "max" : 217137,\n'
                                             '          "mean" : '
                                             '35420.530389,\n'
                                             '          "stddev" : '
                                             '8614.100339,\n'
                                             '          "N" : 1567704\n'
                                             '        },\n'
                                             '        "bw_min" : 894104,\n'
                                             '        "bw_max" : 897792,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895917.571429,\n'
                                             '        "bw_dev" : 1033.548683,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223526,\n'
                                             '        "iops_max" : 224448,\n'
                                             '        "iops_mean" : '
                                             '223979.500000,\n'
                                             '        "iops_stddev" : '
                                             '258.444087,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.142857,\n'
                                             '      "sys_cpu" : 56.985714,\n'
                                             '      "ctx" : 712649,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.070932,\n'
                                             '        "50" : 92.202383,\n'
                                             '        "100" : 7.466940,\n'
                                             '        "250" : 0.259999,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88843: Tue '
                                             'Oct 25 23:47:41 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6124MiB/7001msec)\n'
                                             '    slat (nsec): min=1152, '
                                             'max=175619, avg=1747.09, '
                                             'stdev=791.06\n'
                                             '    clat (usec): min=2, max=215, '
                                             'avg=33.67, stdev= 8.68\n'
                                             '     lat (usec): min=15, '
                                             'max=217, avg=35.42, stdev= 8.61\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   30], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  159]\n'
                                             '   bw (  KiB/s): min=894104, '
                                             'max=897792, per=100.00%, '
                                             'avg=895917.57, stdev=1033.55, '
                                             'samples=14\n'
                                             '   iops        : min=223526, '
                                             'max=224448, avg=223979.50, '
                                             'stdev=258.44, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.07%, 50=92.20%, '
                                             '100=7.47%\n'
                                             '  lat (usec)   : 250=0.26%\n'
                                             '  cpu          : usr=17.14%, '
                                             'sys=56.99%, ctx=712649, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567697,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6124MiB '
                                             '(6421MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.output'),
                                   'state': {'begin': 1666727371.7245438,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7898790836334229,
                                             'end': 1666727372.514423,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_309.state')},
                       'cmd_31': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.output'),
                                  'state': {'begin': 1666726042.0663688,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7834937572479248,
                                            'end': 1666726042.8498626,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_31.state')},
                       'cmd_310': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.output'),
                                   'state': {'begin': 1666727373.5050867,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7769813537597656,
                                             'end': 1666727374.282068,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_310.state')},
                       'cmd_311': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.output'),
                                   'state': {'begin': 1666727374.2836246,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.051728963851929,
                                             'end': 1666727385.3353536,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_311.state')},
                       'cmd_312': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734474,\n'
                                             '  "timestamp_ms" : '
                                             '1666734474814,\n'
                                             '  "time" : "Tue Oct 25 23:47:54 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6416293888,\n'
                                             '        "io_kbytes" : 6265912,\n'
                                             '        "bw_bytes" : 916482486,\n'
                                             '        "bw" : 895002,\n'
                                             '        "iops" : 223749.607199,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566471,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1192,\n'
                                             '          "max" : 40917,\n'
                                             '          "mean" : 1637.510650,\n'
                                             '          "stddev" : '
                                             '539.612945,\n'
                                             '          "N" : 1566471\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 221,\n'
                                             '          "max" : 212459,\n'
                                             '          "mean" : '
                                             '33807.519823,\n'
                                             '          "stddev" : '
                                             '8996.658594,\n'
                                             '          "N" : 1566478,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23936,\n'
                                             '            "5.000000" : 27520,\n'
                                             '            "10.000000" : '
                                             '28800,\n'
                                             '            "20.000000" : '
                                             '29568,\n'
                                             '            "30.000000" : '
                                             '30592,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32640,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '68096,\n'
                                             '            "99.900000" : '
                                             '134144,\n'
                                             '            "99.950000" : '
                                             '144384,\n'
                                             '            "99.990000" : '
                                             '160768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14056,\n'
                                             '          "max" : 213731,\n'
                                             '          "mean" : '
                                             '35445.030097,\n'
                                             '          "stddev" : '
                                             '8951.929801,\n'
                                             '          "N" : 1566478\n'
                                             '        },\n'
                                             '        "bw_min" : 888056,\n'
                                             '        "bw_max" : 897410,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895350.500000,\n'
                                             '        "bw_dev" : 2293.493498,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 222014,\n'
                                             '        "iops_max" : 224352,\n'
                                             '        "iops_mean" : '
                                             '223837.571429,\n'
                                             '        "iops_stddev" : '
                                             '573.327228,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.571429,\n'
                                             '      "sys_cpu" : 55.157143,\n'
                                             '      "ctx" : 696395,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.234029,\n'
                                             '        "50" : 91.735755,\n'
                                             '        "100" : 7.758267,\n'
                                             '        "250" : 0.272204,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88881: Tue '
                                             'Oct 25 23:47:54 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6119MiB/7001msec)\n'
                                             '    slat (nsec): min=1192, '
                                             'max=40917, avg=1637.51, '
                                             'stdev=539.61\n'
                                             '    clat (nsec): min=221, '
                                             'max=212459, avg=33807.52, '
                                             'stdev=8996.66\n'
                                             '     lat (usec): min=14, '
                                             'max=213, avg=35.45, stdev= 8.95\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   28], 10.00th=[   29], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   53],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   69], 99.90th=[  '
                                             '135], 99.95th=[  145],\n'
                                             '     | 99.99th=[  161]\n'
                                             '   bw (  KiB/s): min=888056, '
                                             'max=897410, per=100.00%, '
                                             'avg=895350.50, stdev=2293.49, '
                                             'samples=14\n'
                                             '   iops        : min=222014, '
                                             'max=224352, avg=223837.57, '
                                             'stdev=573.33, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.23%, 50=91.74%, '
                                             '100=7.76%\n'
                                             '  lat (usec)   : 250=0.27%\n'
                                             '  cpu          : usr=17.57%, '
                                             'sys=55.16%, ctx=696395, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566471,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6119MiB '
                                             '(6416MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.output'),
                                   'state': {'begin': 1666727385.3380225,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7821457386016846,
                                             'end': 1666727386.1201682,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_312.state')},
                       'cmd_313': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.output'),
                                   'state': {'begin': 1666727387.1176753,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7792088985443115,
                                             'end': 1666727387.8968842,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_313.state')},
                       'cmd_314': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.output'),
                                   'state': {'begin': 1666727387.898326,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring_cmd" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/ng0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.055521011352539,
                                             'end': 1666727398.953847,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_314.state')},
                       'cmd_315': {'output': 'io_uring_cmd-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734488,\n'
                                             '  "timestamp_ms" : '
                                             '1666734488432,\n'
                                             '  "time" : "Tue Oct 25 23:48:08 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring_cmd",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/ng0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6412873728,\n'
                                             '        "io_kbytes" : 6262572,\n'
                                             '        "bw_bytes" : 915993962,\n'
                                             '        "bw" : 894525,\n'
                                             '        "iops" : 223630.195686,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1565635,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1132,\n'
                                             '          "max" : 29926,\n'
                                             '          "mean" : 1678.465054,\n'
                                             '          "stddev" : '
                                             '744.939201,\n'
                                             '          "N" : 1565636\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 7093,\n'
                                             '          "max" : 395762,\n'
                                             '          "mean" : '
                                             '33756.134564,\n'
                                             '          "stddev" : '
                                             '8803.715628,\n'
                                             '          "N" : 1565642,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24960,\n'
                                             '            "5.000000" : 28800,\n'
                                             '            "10.000000" : '
                                             '29056,\n'
                                             '            "20.000000" : '
                                             '30080,\n'
                                             '            "30.000000" : '
                                             '30592,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '32128,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45824,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '68096,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15138,\n'
                                             '          "max" : 398527,\n'
                                             '          "mean" : '
                                             '35434.600653,\n'
                                             '          "stddev" : '
                                             '8758.561285,\n'
                                             '          "N" : 1565642\n'
                                             '        },\n'
                                             '        "bw_min" : 888520,\n'
                                             '        "bw_max" : 898120,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '894740.571429,\n'
                                             '        "bw_dev" : 2660.843987,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 222130,\n'
                                             '        "iops_max" : 224530,\n'
                                             '        "iops_mean" : '
                                             '223685.142857,\n'
                                             '        "iops_stddev" : '
                                             '665.288700,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.742857,\n'
                                             '      "sys_cpu" : 55.314286,\n'
                                             '      "ctx" : 725537,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.082842,\n'
                                             '        "50" : 92.137439,\n'
                                             '        "100" : 7.499130,\n'
                                             '        "250" : 0.280461,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=88919: Tue '
                                             'Oct 25 23:48:08 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6116MiB/7001msec)\n'
                                             '    slat (nsec): min=1132, '
                                             'max=29926, avg=1678.47, '
                                             'stdev=744.94\n'
                                             '    clat (usec): min=7, max=395, '
                                             'avg=33.76, stdev= 8.80\n'
                                             '     lat (usec): min=15, '
                                             'max=398, avg=35.43, stdev= 8.76\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   30], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   69], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=888520, '
                                             'max=898120, per=100.00%, '
                                             'avg=894740.57, stdev=2660.84, '
                                             'samples=14\n'
                                             '   iops        : min=222130, '
                                             'max=224530, avg=223685.14, '
                                             'stdev=665.29, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.08%, 50=92.14%, 100=7.50%, '
                                             '250=0.28%\n'
                                             '  lat (usec)   : 500=0.01%\n'
                                             '  cpu          : usr=17.74%, '
                                             'sys=55.31%, ctx=725537, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1565635,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6116MiB '
                                             '(6413MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.output'),
                                   'state': {'begin': 1666727398.9567535,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7833278179168701,
                                             'end': 1666727399.7400813,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_315.state')},
                       'cmd_316': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.output'),
                                   'state': {'begin': 1666727401.7916675,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.787926435470581,
                                             'end': 1666727402.579594,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_316.state')},
                       'cmd_317': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.output'),
                                   'state': {'begin': 1666727402.58144,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.487804651260376,
                                             'end': 1666727414.0692446,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_317.state')},
                       'cmd_318': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734503,\n'
                                             '  "timestamp_ms" : '
                                             '1666734503475,\n'
                                             '  "time" : "Tue Oct 25 23:48:23 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6421094400,\n'
                                             '        "io_kbytes" : 6270600,\n'
                                             '        "bw_bytes" : 917168175,\n'
                                             '        "bw" : 895672,\n'
                                             '        "iops" : 223917.011855,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567643,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 661,\n'
                                             '          "max" : 38231,\n'
                                             '          "mean" : 991.606545,\n'
                                             '          "stddev" : '
                                             '179.577583,\n'
                                             '          "N" : 1567643\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1854,\n'
                                             '          "max" : 230002,\n'
                                             '          "mean" : '
                                             '34445.715295,\n'
                                             '          "stddev" : '
                                             '8475.141091,\n'
                                             '          "N" : 1567649,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 28032,\n'
                                             '            "5.000000" : 30080,\n'
                                             '            "10.000000" : '
                                             '30592,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31360,\n'
                                             '            "40.000000" : '
                                             '31616,\n'
                                             '            "50.000000" : '
                                             '31872,\n'
                                             '            "60.000000" : '
                                             '32384,\n'
                                             '            "70.000000" : '
                                             '32640,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '44288,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13204,\n'
                                             '          "max" : 230923,\n'
                                             '          "mean" : '
                                             '35437.321713,\n'
                                             '          "stddev" : '
                                             '8475.876232,\n'
                                             '          "N" : 1567649\n'
                                             '        },\n'
                                             '        "bw_min" : 893891,\n'
                                             '        "bw_max" : 897088,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895900.214286,\n'
                                             '        "bw_dev" : 990.139709,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223472,\n'
                                             '        "iops_max" : 224272,\n'
                                             '        "iops_mean" : '
                                             '223975.000000,\n'
                                             '        "iops_stddev" : '
                                             '247.652051,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 18,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 0.720955,\n'
                                             '        "8" : 99.261694,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.995006,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.022008,\n'
                                             '        "50" : 91.171332,\n'
                                             '        "100" : 8.551245,\n'
                                             '        "250" : 0.255160,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=88970: Tue Oct 25 23:48:23 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6124MiB/7001msec)\n'
                                             '    slat (nsec): min=661, '
                                             'max=38231, avg=991.61, '
                                             'stdev=179.58\n'
                                             '    clat (nsec): min=1854, '
                                             'max=230002, avg=34445.72, '
                                             'stdev=8475.14\n'
                                             '     lat (usec): min=13, '
                                             'max=230, avg=35.44, stdev= 8.48\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   29],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '45], 95.00th=[   53],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893891, '
                                             'max=897088, per=100.00%, '
                                             'avg=895900.21, stdev=990.14, '
                                             'samples=14\n'
                                             '   iops        : min=223472, '
                                             'max=224272, avg=223975.00, '
                                             'stdev=247.65, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '4=0.01%, 10=0.01%, 20=0.02%, '
                                             '50=91.17%\n'
                                             '  lat (usec)   : 100=8.55%, '
                                             '250=0.26%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=18, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=0.7%, 8=99.3%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567643,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6124MiB '
                                             '(6421MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.output'),
                                   'state': {'begin': 1666727414.0718408,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7869017124176025,
                                             'end': 1666727414.8587425,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_318.state')},
                       'cmd_319': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.output'),
                                   'state': {'begin': 1666727416.8832607,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7864069938659668,
                                             'end': 1666727417.6696677,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_319.state')},
                       'cmd_32': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.output'),
                                  'state': {'begin': 1666726042.852027,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-reference_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.04966115951538,
                                            'end': 1666726053.901688,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_32.state')},
                       'cmd_320': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.output'),
                                   'state': {'begin': 1666727417.6715865,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.484341621398926,
                                             'end': 1666727429.1559281,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_320.state')},
                       'cmd_321': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734518,\n'
                                             '  "timestamp_ms" : '
                                             '1666734518563,\n'
                                             '  "time" : "Tue Oct 25 23:48:38 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6416740352,\n'
                                             '        "io_kbytes" : 6266348,\n'
                                             '        "bw_bytes" : 916546257,\n'
                                             '        "bw" : 895064,\n'
                                             '        "iops" : 223765.033567,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566579,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 10229,\n'
                                             '          "mean" : 922.635981,\n'
                                             '          "stddev" : 74.486330,\n'
                                             '          "N" : 1566580\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 13977,\n'
                                             '          "max" : 229741,\n'
                                             '          "mean" : '
                                             '34515.006953,\n'
                                             '          "stddev" : '
                                             '8893.514425,\n'
                                             '          "N" : 1566582,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24192,\n'
                                             '            "5.000000" : 30080,\n'
                                             '            "10.000000" : '
                                             '30336,\n'
                                             '            "20.000000" : '
                                             '30592,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31872,\n'
                                             '            "50.000000" : '
                                             '32384,\n'
                                             '            "60.000000" : '
                                             '32640,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '63744,\n'
                                             '            "99.500000" : '
                                             '69120,\n'
                                             '            "99.900000" : '
                                             '136192,\n'
                                             '            "99.950000" : '
                                             '144384,\n'
                                             '            "99.990000" : '
                                             '160768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14748,\n'
                                             '          "max" : 230652,\n'
                                             '          "mean" : '
                                             '35437.642907,\n'
                                             '          "stddev" : '
                                             '8893.112545,\n'
                                             '          "N" : 1566582\n'
                                             '        },\n'
                                             '        "bw_min" : 892728,\n'
                                             '        "bw_max" : 898024,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895313.500000,\n'
                                             '        "bw_dev" : 1341.825436,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223182,\n'
                                             '        "iops_max" : 224506,\n'
                                             '        "iops_mean" : '
                                             '223828.357143,\n'
                                             '        "iops_stddev" : '
                                             '335.467895,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 0.602140,\n'
                                             '        "8" : 99.388540,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.995614,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.048769,\n'
                                             '        "50" : 91.253298,\n'
                                             '        "100" : 8.411577,\n'
                                             '        "250" : 0.286548,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=89020: Tue Oct 25 23:48:38 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6119MiB/7001msec)\n'
                                             '    slat (nsec): min=651, '
                                             'max=10229, avg=922.64, '
                                             'stdev=74.49\n'
                                             '    clat (usec): min=13, '
                                             'max=229, avg=34.52, stdev= 8.89\n'
                                             '     lat (usec): min=14, '
                                             'max=230, avg=35.44, stdev= 8.89\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '33], 60.00th=[   33],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   53],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   70], 99.90th=[  '
                                             '137], 99.95th=[  145],\n'
                                             '     | 99.99th=[  161]\n'
                                             '   bw (  KiB/s): min=892728, '
                                             'max=898024, per=100.00%, '
                                             'avg=895313.50, stdev=1341.83, '
                                             'samples=14\n'
                                             '   iops        : min=223182, '
                                             'max=224506, avg=223828.36, '
                                             'stdev=335.47, samples=14\n'
                                             '  lat (usec)   : 20=0.05%, '
                                             '50=91.25%, 100=8.41%, 250=0.29%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.1%, '
                                             '4=0.6%, 8=99.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566579,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6119MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.output'),
                                   'state': {'begin': 1666727429.158875,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7881069183349609,
                                             'end': 1666727429.946982,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_321.state')},
                       'cmd_322': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.output'),
                                   'state': {'begin': 1666727431.9935048,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7856662273406982,
                                             'end': 1666727432.779171,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_322.state')},
                       'cmd_323': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.output'),
                                   'state': {'begin': 1666727432.7809138,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring_cmd-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="0" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.463064908981323,
                                             'end': 1666727444.2439787,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_323.state')},
                       'cmd_324': {'output': 'io_uring_cmd-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734533,\n'
                                             '  "timestamp_ms" : '
                                             '1666734533650,\n'
                                             '  "time" : "Tue Oct 25 23:48:53 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring_cmd-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "0",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419181568,\n'
                                             '        "io_kbytes" : 6268732,\n'
                                             '        "bw_bytes" : 916894953,\n'
                                             '        "bw" : 895405,\n'
                                             '        "iops" : 223850.307099,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567176,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 651,\n'
                                             '          "max" : 7023,\n'
                                             '          "mean" : 994.223934,\n'
                                             '          "stddev" : '
                                             '136.149078,\n'
                                             '          "N" : 1567176\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 11421,\n'
                                             '          "max" : 178565,\n'
                                             '          "mean" : '
                                             '34501.146111,\n'
                                             '          "stddev" : '
                                             '8517.845352,\n'
                                             '          "N" : 1567183,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 28544,\n'
                                             '            "5.000000" : 30080,\n'
                                             '            "10.000000" : '
                                             '30592,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31616,\n'
                                             '            "40.000000" : '
                                             '31872,\n'
                                             '            "50.000000" : '
                                             '31872,\n'
                                             '            "60.000000" : '
                                             '32384,\n'
                                             '            "70.000000" : '
                                             '32640,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '44288,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '63232,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '154624\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 12323,\n'
                                             '          "max" : 179727,\n'
                                             '          "mean" : '
                                             '35495.369933,\n'
                                             '          "stddev" : '
                                             '8518.963941,\n'
                                             '          "N" : 1567183\n'
                                             '        },\n'
                                             '        "bw_min" : 892608,\n'
                                             '        "bw_max" : 897816,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895630.214286,\n'
                                             '        "bw_dev" : 1342.108684,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223152,\n'
                                             '        "iops_max" : 224454,\n'
                                             '        "iops_mean" : '
                                             '223907.500000,\n'
                                             '        "iops_stddev" : '
                                             '335.521351,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 99.985714,\n'
                                             '      "sys_cpu" : 0.000000,\n'
                                             '      "ctx" : 4,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 0.630050,\n'
                                             '        "8" : 99.368801,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.997888,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.013847,\n'
                                             '        "50" : 91.109358,\n'
                                             '        "100" : 8.605415,\n'
                                             '        "250" : 0.271827,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring_cmd-bdev_xnvme: '
                                             '(groupid=0, jobs=1): err= 0: '
                                             'pid=89070: Tue Oct 25 23:48:53 '
                                             '2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=651, '
                                             'max=7023, avg=994.22, '
                                             'stdev=136.15\n'
                                             '    clat (usec): min=11, '
                                             'max=178, avg=34.50, stdev= 8.52\n'
                                             '     lat (usec): min=12, '
                                             'max=179, avg=35.50, stdev= 8.52\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   29],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '45], 95.00th=[   53],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  155]\n'
                                             '   bw (  KiB/s): min=892608, '
                                             'max=897816, per=100.00%, '
                                             'avg=895630.21, stdev=1342.11, '
                                             'samples=14\n'
                                             '   iops        : min=223152, '
                                             'max=224454, avg=223907.50, '
                                             'stdev=335.52, samples=14\n'
                                             '  lat (usec)   : 20=0.01%, '
                                             '50=91.11%, 100=8.61%, 250=0.27%\n'
                                             '  cpu          : usr=99.99%, '
                                             'sys=0.00%, ctx=4, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.1%, '
                                             '4=0.6%, 8=99.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567176,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.output'),
                                   'state': {'begin': 1666727444.2467465,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7881875038146973,
                                             'end': 1666727445.034934,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_324.state')},
                       'cmd_325': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.output'),
                                   'state': {'begin': 1666727446.0708587,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7909603118896484,
                                             'end': 1666727446.861819,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_325.state')},
                       'cmd_326': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.output'),
                                   'state': {'begin': 1666727446.8634963,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.056450843811035,
                                             'end': 1666727457.9199471,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_326.state')},
                       'cmd_327': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734547,\n'
                                             '  "timestamp_ms" : '
                                             '1666734547398,\n'
                                             '  "time" : "Tue Oct 25 23:49:07 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418763776,\n'
                                             '        "io_kbytes" : 6268324,\n'
                                             '        "bw_bytes" : 916835277,\n'
                                             '        "bw" : 895346,\n'
                                             '        "iops" : 223835.594915,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567073,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1222,\n'
                                             '          "max" : 43772,\n'
                                             '          "mean" : 1732.981963,\n'
                                             '          "stddev" : '
                                             '738.776513,\n'
                                             '          "N" : 1567074\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 80,\n'
                                             '          "max" : 210004,\n'
                                             '          "mean" : '
                                             '33654.755580,\n'
                                             '          "stddev" : '
                                             '8683.484113,\n'
                                             '          "N" : 1567080,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 28544,\n'
                                             '            "10.000000" : '
                                             '28800,\n'
                                             '            "20.000000" : '
                                             '29568,\n'
                                             '            "30.000000" : '
                                             '30592,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31872,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '128512,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15339,\n'
                                             '          "max" : 211277,\n'
                                             '          "mean" : '
                                             '35387.737334,\n'
                                             '          "stddev" : '
                                             '8632.811278,\n'
                                             '          "N" : 1567080\n'
                                             '        },\n'
                                             '        "bw_min" : 892488,\n'
                                             '        "bw_max" : 898184,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895707.214286,\n'
                                             '        "bw_dev" : 1505.889371,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223122,\n'
                                             '        "iops_max" : 224546,\n'
                                             '        "iops_mean" : '
                                             '223926.928571,\n'
                                             '        "iops_stddev" : '
                                             '376.422525,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.885714,\n'
                                             '      "sys_cpu" : 58.414286,\n'
                                             '      "ctx" : 669792,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.010000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.103697,\n'
                                             '        "50" : 92.161693,\n'
                                             '        "100" : 7.477826,\n'
                                             '        "250" : 0.257040,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2235843,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 70932,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 70932,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89109: Tue '
                                             'Oct 25 23:49:07 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6121MiB/7001msec)\n'
                                             '    slat (nsec): min=1222, '
                                             'max=43772, avg=1732.98, '
                                             'stdev=738.78\n'
                                             '    clat (nsec): min=80, '
                                             'max=210004, avg=33654.76, '
                                             'stdev=8683.48\n'
                                             '     lat (usec): min=15, '
                                             'max=211, avg=35.39, stdev= 8.63\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '129], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892488, '
                                             'max=898184, per=100.00%, '
                                             'avg=895707.21, stdev=1505.89, '
                                             'samples=14\n'
                                             '   iops        : min=223122, '
                                             'max=224546, avg=223926.93, '
                                             'stdev=376.42, samples=14\n'
                                             '  lat (nsec)   : 100=0.01%\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.10%, 50=92.16%, 100=7.48%, '
                                             '250=0.26%\n'
                                             '  cpu          : usr=16.89%, '
                                             'sys=58.41%, ctx=669792, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567073,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6121MiB '
                                             '(6419MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2235843/0, '
                                             'merge=0/0, ticks=70932/0, '
                                             'in_queue=70932, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.output'),
                                   'state': {'begin': 1666727457.9227653,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7953412532806396,
                                             'end': 1666727458.7181065,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_327.state')},
                       'cmd_328': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.output'),
                                   'state': {'begin': 1666727459.7156532,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7870519161224365,
                                             'end': 1666727460.502705,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_328.state')},
                       'cmd_329': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.output'),
                                   'state': {'begin': 1666727460.504661,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.054227828979492,
                                             'end': 1666727471.558889,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_329.state')},
                       'cmd_33': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733143,\n'
                                            '  "timestamp_ms" : '
                                            '1666733143402,\n'
                                            '  "time" : "Tue Oct 25 23:25:43 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1824374784,\n'
                                            '        "io_kbytes" : 1781616,\n'
                                            '        "bw_bytes" : 260587742,\n'
                                            '        "bw" : 254480,\n'
                                            '        "iops" : 63620.054278,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 445404,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2424,\n'
                                            '          "max" : 34645,\n'
                                            '          "mean" : 2592.826618,\n'
                                            '          "stddev" : 222.766188,\n'
                                            '          "N" : 445404\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 240,\n'
                                            '          "max" : 122219,\n'
                                            '          "mean" : 12494.563899,\n'
                                            '          "stddev" : '
                                            '2015.777434,\n'
                                            '          "N" : 445404,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 11968,\n'
                                            '            "10.000000" : 11968,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12096,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12224,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12480,\n'
                                            '            "99.000000" : 22400,\n'
                                            '            "99.500000" : 27776,\n'
                                            '            "99.900000" : 33024,\n'
                                            '            "99.950000" : 49920,\n'
                                            '            "99.990000" : 64768\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12434,\n'
                                            '          "max" : 124784,\n'
                                            '          "mean" : 15087.390517,\n'
                                            '          "stddev" : '
                                            '2032.769095,\n'
                                            '          "N" : 445404\n'
                                            '        },\n'
                                            '        "bw_min" : 253272,\n'
                                            '        "bw_max" : 255968,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '254578.357143,\n'
                                            '        "bw_dev" : 701.625103,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 63318,\n'
                                            '        "iops_max" : 63992,\n'
                                            '        "iops_mean" : '
                                            '63644.571429,\n'
                                            '        "iops_stddev" : '
                                            '175.405778,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.200000,\n'
                                            '      "sys_cpu" : 36.728571,\n'
                                            '      "ctx" : 445388,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.600821,\n'
                                            '        "50" : 1.339009,\n'
                                            '        "100" : 0.052537,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 638282,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 6018,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 6018,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=84810: Tue '
                                            'Oct 25 23:25:43 2022\n'
                                            '  read: IOPS=63.6k, BW=249MiB/s '
                                            '(261MB/s)(1740MiB/7001msec)\n'
                                            '    slat (nsec): min=2424, '
                                            'max=34645, avg=2592.83, '
                                            'stdev=222.77\n'
                                            '    clat (nsec): min=240, '
                                            'max=122219, avg=12494.56, '
                                            'stdev=2015.78\n'
                                            '     lat (usec): min=12, max=124, '
                                            'avg=15.09, stdev= 2.03\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[11968], 10.00th=[11968], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12096], '
                                            '40.00th=[12224], 50.00th=[12224], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12480], '
                                            '95.00th=[12480],\n'
                                            '     | 99.00th=[22400], '
                                            '99.50th=[27776], 99.90th=[33024], '
                                            '99.95th=[49920],\n'
                                            '     | 99.99th=[64768]\n'
                                            '   bw (  KiB/s): min=253272, '
                                            'max=255968, per=100.00%, '
                                            'avg=254578.36, stdev=701.63, '
                                            'samples=14\n'
                                            '   iops        : min=63318, '
                                            'max=63992, avg=63644.57, '
                                            'stdev=175.41, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.60%, 50=1.34%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=11.20%, '
                                            'sys=36.73%, ctx=445388, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=445404,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=249MiB/s (261MB/s), '
                                            '249MiB/s-249MiB/s '
                                            '(261MB/s-261MB/s), io=1740MiB '
                                            '(1824MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=638282/0, '
                                            'merge=0/0, ticks=6018/0, '
                                            'in_queue=6018, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.output'),
                                  'state': {'begin': 1666726053.9044824,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7907357215881348,
                                            'end': 1666726054.695218,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_33.state')},
                       'cmd_330': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734561,\n'
                                             '  "timestamp_ms" : '
                                             '1666734561035,\n'
                                             '  "time" : "Tue Oct 25 23:49:21 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418571264,\n'
                                             '        "io_kbytes" : 6268136,\n'
                                             '        "bw_bytes" : 916807779,\n'
                                             '        "bw" : 895320,\n'
                                             '        "iops" : 223828.881588,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567026,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1212,\n'
                                             '          "max" : 36078,\n'
                                             '          "mean" : 1714.383513,\n'
                                             '          "stddev" : '
                                             '722.179076,\n'
                                             '          "N" : 1567027\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 12804,\n'
                                             '          "max" : 206107,\n'
                                             '          "mean" : '
                                             '33675.974473,\n'
                                             '          "stddev" : '
                                             '8691.726209,\n'
                                             '          "N" : 1567033,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24960,\n'
                                             '            "5.000000" : 28544,\n'
                                             '            "10.000000" : '
                                             '28800,\n'
                                             '            "20.000000" : '
                                             '29824,\n'
                                             '            "30.000000" : '
                                             '30592,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '128512,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15399,\n'
                                             '          "max" : 207379,\n'
                                             '          "mean" : '
                                             '35390.359061,\n'
                                             '          "stddev" : '
                                             '8641.981671,\n'
                                             '          "N" : 1567033\n'
                                             '        },\n'
                                             '        "bw_min" : 893378,\n'
                                             '        "bw_max" : 897136,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895682.857143,\n'
                                             '        "bw_dev" : 1185.473801,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223344,\n'
                                             '        "iops_max" : 224284,\n'
                                             '        "iops_mean" : '
                                             '223920.642857,\n'
                                             '        "iops_stddev" : '
                                             '296.455888,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.600000,\n'
                                             '      "sys_cpu" : 58.057143,\n'
                                             '      "ctx" : 693034,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.086406,\n'
                                             '        "50" : 92.191195,\n'
                                             '        "100" : 7.458842,\n'
                                             '        "250" : 0.264003,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2235997,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 71144,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 71144,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89147: Tue '
                                             'Oct 25 23:49:21 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6121MiB/7001msec)\n'
                                             '    slat (nsec): min=1212, '
                                             'max=36078, avg=1714.38, '
                                             'stdev=722.18\n'
                                             '    clat (usec): min=12, '
                                             'max=206, avg=33.68, stdev= 8.69\n'
                                             '     lat (usec): min=15, '
                                             'max=207, avg=35.39, stdev= 8.64\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '129], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893378, '
                                             'max=897136, per=100.00%, '
                                             'avg=895682.86, stdev=1185.47, '
                                             'samples=14\n'
                                             '   iops        : min=223344, '
                                             'max=224284, avg=223920.64, '
                                             'stdev=296.46, samples=14\n'
                                             '  lat (usec)   : 20=0.09%, '
                                             '50=92.19%, 100=7.46%, 250=0.26%\n'
                                             '  cpu          : usr=16.60%, '
                                             'sys=58.06%, ctx=693034, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567026,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6121MiB '
                                             '(6419MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2235997/0, '
                                             'merge=0/0, ticks=71144/0, '
                                             'in_queue=71144, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.output'),
                                   'state': {'begin': 1666727471.5615969,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7940971851348877,
                                             'end': 1666727472.355694,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_330.state')},
                       'cmd_331': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.output'),
                                   'state': {'begin': 1666727473.3489244,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7900288105010986,
                                             'end': 1666727474.1389532,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_331.state')},
                       'cmd_332': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.output'),
                                   'state': {'begin': 1666727474.1414618,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="io_uring" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.046249628067017,
                                             'end': 1666727485.1877115,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_332.state')},
                       'cmd_333': {'output': 'io_uring-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=io_uring, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734574,\n'
                                             '  "timestamp_ms" : '
                                             '1666734574665,\n'
                                             '  "time" : "Tue Oct 25 23:49:34 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"io_uring",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419468288,\n'
                                             '        "io_kbytes" : 6269012,\n'
                                             '        "bw_bytes" : 916935907,\n'
                                             '        "bw" : 895445,\n'
                                             '        "iops" : 223860.162834,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567245,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1232,\n'
                                             '          "max" : 53460,\n'
                                             '          "mean" : 1707.207037,\n'
                                             '          "stddev" : '
                                             '717.038812,\n'
                                             '          "N" : 1567246\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 7434,\n'
                                             '          "max" : 196628,\n'
                                             '          "mean" : '
                                             '33672.910071,\n'
                                             '          "stddev" : '
                                             '8730.705101,\n'
                                             '          "N" : 1567252,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 25216,\n'
                                             '            "5.000000" : 28544,\n'
                                             '            "10.000000" : '
                                             '28800,\n'
                                             '            "20.000000" : '
                                             '29312,\n'
                                             '            "30.000000" : '
                                             '30592,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31872,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15579,\n'
                                             '          "max" : 198051,\n'
                                             '          "mean" : '
                                             '35380.117355,\n'
                                             '          "stddev" : '
                                             '8681.842147,\n'
                                             '          "N" : 1567252\n'
                                             '        },\n'
                                             '        "bw_min" : 892529,\n'
                                             '        "bw_max" : 897192,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895805.071429,\n'
                                             '        "bw_dev" : 1222.038050,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223132,\n'
                                             '        "iops_max" : 224298,\n'
                                             '        "iops_mean" : '
                                             '223951.357143,\n'
                                             '        "iops_stddev" : '
                                             '305.643233,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 17.300000,\n'
                                             '      "sys_cpu" : 57.928571,\n'
                                             '      "ctx" : 667803,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.066550,\n'
                                             '        "50" : 92.209706,\n'
                                             '        "100" : 7.450845,\n'
                                             '        "250" : 0.273154,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2236081,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 71107,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 71107,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89185: Tue '
                                             'Oct 25 23:49:34 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=1232, '
                                             'max=53460, avg=1707.21, '
                                             'stdev=717.04\n'
                                             '    clat (usec): min=7, max=196, '
                                             'avg=33.67, stdev= 8.73\n'
                                             '     lat (usec): min=15, '
                                             'max=198, avg=35.38, stdev= 8.68\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   26],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892529, '
                                             'max=897192, per=100.00%, '
                                             'avg=895805.07, stdev=1222.04, '
                                             'samples=14\n'
                                             '   iops        : min=223132, '
                                             'max=224298, avg=223951.36, '
                                             'stdev=305.64, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.07%, 50=92.21%, 100=7.45%, '
                                             '250=0.27%\n'
                                             '  cpu          : usr=17.30%, '
                                             'sys=57.93%, ctx=667803, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567245,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6419MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2236081/0, '
                                             'merge=0/0, ticks=71107/0, '
                                             'in_queue=71107, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.output'),
                                   'state': {'begin': 1666727485.1905935,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7888400554656982,
                                             'end': 1666727485.9794335,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_333.state')},
                       'cmd_334': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.output'),
                                   'state': {'begin': 1666727488.0161452,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7870473861694336,
                                             'end': 1666727488.8031926,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_334.state')},
                       'cmd_335': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.output'),
                                   'state': {'begin': 1666727488.8052003,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.470963478088379,
                                             'end': 1666727500.2761638,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_335.state')},
                       'cmd_336': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734589,\n'
                                             '  "timestamp_ms" : '
                                             '1666734589681,\n'
                                             '  "time" : "Tue Oct 25 23:49:49 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419386368,\n'
                                             '        "io_kbytes" : 6268932,\n'
                                             '        "bw_bytes" : 916924206,\n'
                                             '        "bw" : 895433,\n'
                                             '        "iops" : 223857.448936,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567226,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 17783,\n'
                                             '          "mean" : 177.147500,\n'
                                             '          "stddev" : 78.228730,\n'
                                             '          "N" : 1567226\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 13115,\n'
                                             '          "max" : 1356474,\n'
                                             '          "mean" : '
                                             '35300.071299,\n'
                                             '          "stddev" : '
                                             '8917.372350,\n'
                                             '          "N" : 1567233,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 29568,\n'
                                             '            "5.000000" : 32384,\n'
                                             '            "10.000000" : '
                                             '32384,\n'
                                             '            "20.000000" : '
                                             '32640,\n'
                                             '            "30.000000" : '
                                             '32640,\n'
                                             '            "40.000000" : '
                                             '32640,\n'
                                             '            "50.000000" : '
                                             '32640,\n'
                                             '            "60.000000" : '
                                             '32640,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '53504,\n'
                                             '            "99.000000" : '
                                             '63744,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13395,\n'
                                             '          "max" : 1356644,\n'
                                             '          "mean" : '
                                             '35477.218807,\n'
                                             '          "stddev" : '
                                             '8916.741132,\n'
                                             '          "N" : 1567233\n'
                                             '        },\n'
                                             '        "bw_min" : 893360,\n'
                                             '        "bw_max" : 897640,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895644.571429,\n'
                                             '        "bw_dev" : 1100.192830,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223340,\n'
                                             '        "iops_max" : 224410,\n'
                                             '        "iops_mean" : '
                                             '223911.142857,\n'
                                             '        "iops_stddev" : '
                                             '275.048208,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.585714,\n'
                                             '      "sys_cpu" : 60.385714,\n'
                                             '      "ctx" : 40,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.291773,\n'
                                             '        "8" : 98.704335,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.996949,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.010000,\n'
                                             '        "50" : 90.676265,\n'
                                             '        "100" : 9.055108,\n'
                                             '        "250" : 0.262374,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=89233: Tue '
                                             'Oct 25 23:49:49 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=17783, avg=177.15, '
                                             'stdev=78.23\n'
                                             '    clat (usec): min=13, '
                                             'max=1356, avg=35.30, stdev= '
                                             '8.92\n'
                                             '     lat (usec): min=13, '
                                             'max=1356, avg=35.48, stdev= '
                                             '8.92\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   30],  '
                                             '5.00th=[   33], 10.00th=[   33], '
                                             '20.00th=[   33],\n'
                                             '     | 30.00th=[   33], '
                                             '40.00th=[   33], 50.00th=[   '
                                             '33], 60.00th=[   33],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   54],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893360, '
                                             'max=897640, per=100.00%, '
                                             'avg=895644.57, stdev=1100.19, '
                                             'samples=14\n'
                                             '   iops        : min=223340, '
                                             'max=224410, avg=223911.14, '
                                             'stdev=275.05, samples=14\n'
                                             '  lat (usec)   : 20=0.01%, '
                                             '50=90.68%, 100=9.06%, 250=0.26%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=39.59%, '
                                             'sys=60.39%, ctx=40, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.3%, 8=98.7%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567226,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.output'),
                                   'state': {'begin': 1666727500.2788785,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8842198848724365,
                                             'end': 1666727501.1630983,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_336.state')},
                       'cmd_337': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.output'),
                                   'state': {'begin': 1666727503.4680705,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8344917297363281,
                                             'end': 1666727504.3025622,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_337.state')},
                       'cmd_338': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.output'),
                                   'state': {'begin': 1666727504.3044047,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.516225814819336,
                                             'end': 1666727515.8206306,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_338.state')},
                       'cmd_339': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734605,\n'
                                             '  "timestamp_ms" : '
                                             '1666734605226,\n'
                                             '  "time" : "Tue Oct 25 23:50:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418595840,\n'
                                             '        "io_kbytes" : 6268160,\n'
                                             '        "bw_bytes" : 916811289,\n'
                                             '        "bw" : 895323,\n'
                                             '        "iops" : 223829.881446,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567033,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 120,\n'
                                             '          "max" : 18484,\n'
                                             '          "mean" : 180.659000,\n'
                                             '          "stddev" : 80.296904,\n'
                                             '          "N" : 1567033\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 13105,\n'
                                             '          "max" : 191649,\n'
                                             '          "mean" : '
                                             '35285.179822,\n'
                                             '          "stddev" : '
                                             '8502.394509,\n'
                                             '          "N" : 1567040,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 29824,\n'
                                             '            "5.000000" : 32384,\n'
                                             '            "10.000000" : '
                                             '32384,\n'
                                             '            "20.000000" : '
                                             '32640,\n'
                                             '            "30.000000" : '
                                             '32640,\n'
                                             '            "40.000000" : '
                                             '32640,\n'
                                             '            "50.000000" : '
                                             '32640,\n'
                                             '            "60.000000" : '
                                             '32640,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '53504,\n'
                                             '            "99.000000" : '
                                             '63744,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13355,\n'
                                             '          "max" : 191830,\n'
                                             '          "mean" : '
                                             '35465.838813,\n'
                                             '          "stddev" : '
                                             '8501.391034,\n'
                                             '          "N" : 1567040\n'
                                             '        },\n'
                                             '        "bw_min" : 891984,\n'
                                             '        "bw_max" : 898280,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895534.142857,\n'
                                             '        "bw_dev" : 1537.441324,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 222996,\n'
                                             '        "iops_max" : 224570,\n'
                                             '        "iops_mean" : '
                                             '223883.500000,\n'
                                             '        "iops_stddev" : '
                                             '384.365761,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 42.957143,\n'
                                             '      "sys_cpu" : 57.014286,\n'
                                             '      "ctx" : 11,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.244645,\n'
                                             '        "8" : 98.750441,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.996629,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.010000,\n'
                                             '        "50" : 90.670713,\n'
                                             '        "100" : 9.049012,\n'
                                             '        "250" : 0.275425,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=89282: Tue '
                                             'Oct 25 23:50:05 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6121MiB/7001msec)\n'
                                             '    slat (nsec): min=120, '
                                             'max=18484, avg=180.66, '
                                             'stdev=80.30\n'
                                             '    clat (usec): min=13, '
                                             'max=191, avg=35.29, stdev= 8.50\n'
                                             '     lat (usec): min=13, '
                                             'max=191, avg=35.47, stdev= 8.50\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   30],  '
                                             '5.00th=[   33], 10.00th=[   33], '
                                             '20.00th=[   33],\n'
                                             '     | 30.00th=[   33], '
                                             '40.00th=[   33], 50.00th=[   '
                                             '33], 60.00th=[   33],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   54],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=891984, '
                                             'max=898280, per=100.00%, '
                                             'avg=895534.14, stdev=1537.44, '
                                             'samples=14\n'
                                             '   iops        : min=222996, '
                                             'max=224570, avg=223883.50, '
                                             'stdev=384.37, samples=14\n'
                                             '  lat (usec)   : 20=0.01%, '
                                             '50=90.67%, 100=9.05%, 250=0.28%\n'
                                             '  cpu          : usr=42.96%, '
                                             'sys=57.01%, ctx=11, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.2%, 8=98.8%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567033,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6121MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.output'),
                                   'state': {'begin': 1666727515.8234217,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8254802227020264,
                                             'end': 1666727516.648902,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_339.state')},
                       'cmd_34': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.output'),
                                  'state': {'begin': 1666726055.6835701,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7879164218902588,
                                            'end': 1666726056.4714866,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_34.state')},
                       'cmd_340': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.output'),
                                   'state': {'begin': 1666727518.7496185,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.813143253326416,
                                             'end': 1666727519.5627618,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_340.state')},
                       'cmd_341': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.output'),
                                   'state': {'begin': 1666727519.5640981,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_uring" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.459917545318604,
                                             'end': 1666727531.0240157,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_341.state')},
                       'cmd_342': {'output': 'io_uring-bdev_uring: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734620,\n'
                                             '  "timestamp_ms" : '
                                             '1666734620429,\n'
                                             '  "time" : "Tue Oct 25 23:50:20 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_uring",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_uring",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6415040512,\n'
                                             '        "io_kbytes" : 6264688,\n'
                                             '        "bw_bytes" : 916303458,\n'
                                             '        "bw" : 894827,\n'
                                             '        "iops" : 223705.899157,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566165,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 110,\n'
                                             '          "max" : 41308,\n'
                                             '          "mean" : 199.295783,\n'
                                             '          "stddev" : '
                                             '128.404646,\n'
                                             '          "N" : 1566165\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 12804,\n'
                                             '          "max" : 177943,\n'
                                             '          "mean" : '
                                             '35310.666389,\n'
                                             '          "stddev" : '
                                             '8531.519283,\n'
                                             '          "N" : 1566172,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 29568,\n'
                                             '            "5.000000" : 32384,\n'
                                             '            "10.000000" : '
                                             '32384,\n'
                                             '            "20.000000" : '
                                             '32640,\n'
                                             '            "30.000000" : '
                                             '32640,\n'
                                             '            "40.000000" : '
                                             '32640,\n'
                                             '            "50.000000" : '
                                             '32640,\n'
                                             '            "60.000000" : '
                                             '32640,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '33024,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '53504,\n'
                                             '            "99.000000" : '
                                             '63744,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '142336,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13055,\n'
                                             '          "max" : 178154,\n'
                                             '          "mean" : '
                                             '35509.962140,\n'
                                             '          "stddev" : '
                                             '8530.220824,\n'
                                             '          "N" : 1566172\n'
                                             '        },\n'
                                             '        "bw_min" : 893232,\n'
                                             '        "bw_max" : 895976,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895036.428571,\n'
                                             '        "bw_dev" : 819.653277,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223308,\n'
                                             '        "iops_max" : 223994,\n'
                                             '        "iops_mean" : '
                                             '223759.214286,\n'
                                             '        "iops_stddev" : '
                                             '205.000630,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 39.642857,\n'
                                             '      "sys_cpu" : 60.328571,\n'
                                             '      "ctx" : 19,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.353178,\n'
                                             '        "8" : 98.618983,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.991192,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.011110,\n'
                                             '        "50" : 90.645749,\n'
                                             '        "100" : 9.069670,\n'
                                             '        "250" : 0.273917,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_uring: (groupid=0, '
                                             'jobs=1): err= 0: pid=89331: Tue '
                                             'Oct 25 23:50:20 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6118MiB/7001msec)\n'
                                             '    slat (nsec): min=110, '
                                             'max=41308, avg=199.30, '
                                             'stdev=128.40\n'
                                             '    clat (usec): min=12, '
                                             'max=177, avg=35.31, stdev= 8.53\n'
                                             '     lat (usec): min=13, '
                                             'max=178, avg=35.51, stdev= 8.53\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   30],  '
                                             '5.00th=[   33], 10.00th=[   33], '
                                             '20.00th=[   33],\n'
                                             '     | 30.00th=[   33], '
                                             '40.00th=[   33], 50.00th=[   '
                                             '33], 60.00th=[   33],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '46], 95.00th=[   54],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  143],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893232, '
                                             'max=895976, per=100.00%, '
                                             'avg=895036.43, stdev=819.65, '
                                             'samples=14\n'
                                             '   iops        : min=223308, '
                                             'max=223994, avg=223759.21, '
                                             'stdev=205.00, samples=14\n'
                                             '  lat (usec)   : 20=0.01%, '
                                             '50=90.65%, 100=9.07%, 250=0.27%\n'
                                             '  cpu          : usr=39.64%, '
                                             'sys=60.33%, ctx=19, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.4%, 8=98.6%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566165,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6118MiB '
                                             '(6415MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.output'),
                                   'state': {'begin': 1666727531.026178,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7857508659362793,
                                             'end': 1666727531.8119287,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_342.state')},
                       'cmd_343': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.output'),
                                   'state': {'begin': 1666727533.9081874,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7813472747802734,
                                             'end': 1666727534.6895347,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_343.state')},
                       'cmd_344': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.output'),
                                   'state': {'begin': 1666727534.69146,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.462684392929077,
                                             'end': 1666727546.1541443,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_344.state')},
                       'cmd_345': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734635,\n'
                                             '  "timestamp_ms" : '
                                             '1666734635553,\n'
                                             '  "time" : "Tue Oct 25 23:50:35 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6416384000,\n'
                                             '        "io_kbytes" : 6266000,\n'
                                             '        "bw_bytes" : 916495357,\n'
                                             '        "bw" : 895014,\n'
                                             '        "iops" : 223752.749607,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566493,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1482,\n'
                                             '          "max" : 67687,\n'
                                             '          "mean" : 1754.159928,\n'
                                             '          "stddev" : '
                                             '246.058703,\n'
                                             '          "N" : 1566493\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 15098,\n'
                                             '          "max" : 202079,\n'
                                             '          "mean" : '
                                             '33741.697037,\n'
                                             '          "stddev" : '
                                             '8649.734313,\n'
                                             '          "N" : 1566500,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '30848,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31104,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 17352,\n'
                                             '          "max" : 203803,\n'
                                             '          "mean" : '
                                             '35495.856783,\n'
                                             '          "stddev" : '
                                             '8639.780490,\n'
                                             '          "N" : 1566500\n'
                                             '        },\n'
                                             '        "bw_min" : 894056,\n'
                                             '        "bw_max" : 897080,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895219.785714,\n'
                                             '        "bw_dev" : 929.395517,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223514,\n'
                                             '        "iops_max" : 224270,\n'
                                             '        "iops_mean" : '
                                             '223804.785714,\n'
                                             '        "iops_stddev" : '
                                             '232.359198,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 30.385714,\n'
                                             '      "sys_cpu" : 69.585714,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.026684,\n'
                                             '        "50" : 91.094183,\n'
                                             '        "100" : 8.585867,\n'
                                             '        "250" : 0.293713,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89380: Tue '
                                             'Oct 25 23:50:35 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6119MiB/7001msec)\n'
                                             '    slat (nsec): min=1482, '
                                             'max=67687, avg=1754.16, '
                                             'stdev=246.06\n'
                                             '    clat (usec): min=15, '
                                             'max=202, avg=33.74, stdev= 8.65\n'
                                             '     lat (usec): min=17, '
                                             'max=203, avg=35.50, stdev= 8.64\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894056, '
                                             'max=897080, per=100.00%, '
                                             'avg=895219.79, stdev=929.40, '
                                             'samples=14\n'
                                             '   iops        : min=223514, '
                                             'max=224270, avg=223804.79, '
                                             'stdev=232.36, samples=14\n'
                                             '  lat (usec)   : 20=0.03%, '
                                             '50=91.09%, 100=8.59%, 250=0.29%\n'
                                             '  cpu          : usr=30.39%, '
                                             'sys=69.59%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566493,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6119MiB '
                                             '(6416MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.output'),
                                   'state': {'begin': 1666727546.1562865,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8221015930175781,
                                             'end': 1666727546.978388,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_345.state')},
                       'cmd_346': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.output'),
                                   'state': {'begin': 1666727549.0630267,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7824573516845703,
                                             'end': 1666727549.845484,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_346.state')},
                       'cmd_347': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.output'),
                                   'state': {'begin': 1666727549.8478909,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.456165552139282,
                                             'end': 1666727561.3040564,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_347.state')},
                       'cmd_348': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734650,\n'
                                             '  "timestamp_ms" : '
                                             '1666734650706,\n'
                                             '  "time" : "Tue Oct 25 23:50:50 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418513920,\n'
                                             '        "io_kbytes" : 6268080,\n'
                                             '        "bw_bytes" : 916799588,\n'
                                             '        "bw" : 895312,\n'
                                             '        "iops" : 223827.024711,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567013,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1473,\n'
                                             '          "max" : 25368,\n'
                                             '          "mean" : 1749.800455,\n'
                                             '          "stddev" : '
                                             '234.769642,\n'
                                             '          "N" : 1567013\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 12604,\n'
                                             '          "max" : 1083192,\n'
                                             '          "mean" : '
                                             '33745.384779,\n'
                                             '          "stddev" : '
                                             '8890.164459,\n'
                                             '          "N" : 1567020,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '30848,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31104,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14317,\n'
                                             '          "max" : 1084925,\n'
                                             '          "mean" : '
                                             '35495.185114,\n'
                                             '          "stddev" : '
                                             '8880.807151,\n'
                                             '          "N" : 1567020\n'
                                             '        },\n'
                                             '        "bw_min" : 893456,\n'
                                             '        "bw_max" : 896816,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895527.357143,\n'
                                             '        "bw_dev" : 964.990044,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223364,\n'
                                             '        "iops_max" : 224204,\n'
                                             '        "iops_mean" : '
                                             '223881.785714,\n'
                                             '        "iops_stddev" : '
                                             '241.228962,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 25.757143,\n'
                                             '      "sys_cpu" : 74.214286,\n'
                                             '      "ctx" : 42,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.046075,\n'
                                             '        "50" : 91.074356,\n'
                                             '        "100" : 8.607331,\n'
                                             '        "250" : 0.272174,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89429: Tue '
                                             'Oct 25 23:50:50 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6121MiB/7001msec)\n'
                                             '    slat (nsec): min=1473, '
                                             'max=25368, avg=1749.80, '
                                             'stdev=234.77\n'
                                             '    clat (usec): min=12, '
                                             'max=1083, avg=33.75, stdev= '
                                             '8.89\n'
                                             '     lat (usec): min=14, '
                                             'max=1084, avg=35.50, stdev= '
                                             '8.88\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893456, '
                                             'max=896816, per=100.00%, '
                                             'avg=895527.36, stdev=964.99, '
                                             'samples=14\n'
                                             '   iops        : min=223364, '
                                             'max=224204, avg=223881.79, '
                                             'stdev=241.23, samples=14\n'
                                             '  lat (usec)   : 20=0.05%, '
                                             '50=91.07%, 100=8.61%, 250=0.27%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=25.76%, '
                                             'sys=74.21%, ctx=42, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567013,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6121MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.output'),
                                   'state': {'begin': 1666727561.3071404,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7886478900909424,
                                             'end': 1666727562.0957882,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_348.state')},
                       'cmd_349': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.output'),
                                   'state': {'begin': 1666727564.1502848,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7863068580627441,
                                             'end': 1666727564.9365916,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_349.state')},
                       'cmd_35': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.output'),
                                  'state': {'begin': 1666726056.473827,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="io_uring" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-reference_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.052614212036133,
                                            'end': 1666726067.526441,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_35.state')},
                       'cmd_350': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.output'),
                                   'state': {'begin': 1666727564.9434702,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.563092470169067,
                                             'end': 1666727576.5065627,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_350.state')},
                       'cmd_351': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734665,\n'
                                             '  "timestamp_ms" : '
                                             '1666734665909,\n'
                                             '  "time" : "Tue Oct 25 23:51:05 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418640896,\n'
                                             '        "io_kbytes" : 6268204,\n'
                                             '        "bw_bytes" : 916817725,\n'
                                             '        "bw" : 895329,\n'
                                             '        "iops" : 223831.452650,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567044,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1472,\n'
                                             '          "max" : 25949,\n'
                                             '          "mean" : 1754.159593,\n'
                                             '          "stddev" : '
                                             '224.643680,\n'
                                             '          "N" : 1567044\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 15079,\n'
                                             '          "max" : 220213,\n'
                                             '          "mean" : '
                                             '33737.245896,\n'
                                             '          "stddev" : '
                                             '8603.637014,\n'
                                             '          "N" : 1567051,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '30848,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31104,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 17322,\n'
                                             '          "max" : 221956,\n'
                                             '          "mean" : '
                                             '35491.405287,\n'
                                             '          "stddev" : '
                                             '8595.079541,\n'
                                             '          "N" : 1567051\n'
                                             '        },\n'
                                             '        "bw_min" : 892688,\n'
                                             '        "bw_max" : 897683,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895548.785714,\n'
                                             '        "bw_dev" : 1287.687264,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223172,\n'
                                             '        "iops_max" : 224420,\n'
                                             '        "iops_mean" : '
                                             '223887.142857,\n'
                                             '        "iops_stddev" : '
                                             '321.708625,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 29.585714,\n'
                                             '      "sys_cpu" : 70.385714,\n'
                                             '      "ctx" : 13,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.035098,\n'
                                             '        "50" : 91.088827,\n'
                                             '        "100" : 8.597908,\n'
                                             '        "250" : 0.278614,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89479: Tue '
                                             'Oct 25 23:51:05 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6121MiB/7001msec)\n'
                                             '    slat (nsec): min=1472, '
                                             'max=25949, avg=1754.16, '
                                             'stdev=224.64\n'
                                             '    clat (usec): min=15, '
                                             'max=220, avg=33.74, stdev= 8.60\n'
                                             '     lat (usec): min=17, '
                                             'max=221, avg=35.49, stdev= 8.60\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892688, '
                                             'max=897683, per=100.00%, '
                                             'avg=895548.79, stdev=1287.69, '
                                             'samples=14\n'
                                             '   iops        : min=223172, '
                                             'max=224420, avg=223887.14, '
                                             'stdev=321.71, samples=14\n'
                                             '  lat (usec)   : 20=0.04%, '
                                             '50=91.09%, 100=8.60%, 250=0.28%\n'
                                             '  cpu          : usr=29.59%, '
                                             'sys=70.39%, ctx=13, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567044,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6121MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.output'),
                                   'state': {'begin': 1666727576.5091283,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7935681343078613,
                                             'end': 1666727577.3026965,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_351.state')},
                       'cmd_352': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.output'),
                                   'state': {'begin': 1666727578.3296173,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.8199388980865479,
                                             'end': 1666727579.1495562,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_352.state')},
                       'cmd_353': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.output'),
                                   'state': {'begin': 1666727579.1509423,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.05217170715332,
                                             'end': 1666727590.203114,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_353.state')},
                       'cmd_354': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734679,\n'
                                             '  "timestamp_ms" : '
                                             '1666734679679,\n'
                                             '  "time" : "Tue Oct 25 23:51:19 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6417080320,\n'
                                             '        "io_kbytes" : 6266680,\n'
                                             '        "bw_bytes" : 916594817,\n'
                                             '        "bw" : 895112,\n'
                                             '        "iops" : 223777.031853,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566663,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1312,\n'
                                             '          "max" : 178064,\n'
                                             '          "mean" : 2022.793881,\n'
                                             '          "stddev" : '
                                             '860.278382,\n'
                                             '          "N" : 1566663\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 5811,\n'
                                             '          "max" : 218590,\n'
                                             '          "mean" : '
                                             '33429.997546,\n'
                                             '          "stddev" : '
                                             '8920.942316,\n'
                                             '          "N" : 1566670,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23936,\n'
                                             '            "5.000000" : 28032,\n'
                                             '            "10.000000" : '
                                             '28032,\n'
                                             '            "20.000000" : '
                                             '28544,\n'
                                             '            "30.000000" : '
                                             '30336,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31872,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15940,\n'
                                             '          "max" : 221325,\n'
                                             '          "mean" : '
                                             '35452.790505,\n'
                                             '          "stddev" : '
                                             '8811.650507,\n'
                                             '          "N" : 1566670\n'
                                             '        },\n'
                                             '        "bw_min" : 894184,\n'
                                             '        "bw_max" : 897432,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895345.000000,\n'
                                             '        "bw_dev" : 952.534675,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223546,\n'
                                             '        "iops_max" : 224358,\n'
                                             '        "iops_mean" : '
                                             '223836.214286,\n'
                                             '        "iops_stddev" : '
                                             '238.146748,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 16.414286,\n'
                                             '      "sys_cpu" : 62.785714,\n'
                                             '      "ctx" : 539835,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.228958,\n'
                                             '        "50" : 92.130024,\n'
                                             '        "100" : 7.357294,\n'
                                             '        "250" : 0.284107,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89518: Tue '
                                             'Oct 25 23:51:19 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6120MiB/7001msec)\n'
                                             '    slat (nsec): min=1312, '
                                             'max=178064, avg=2022.79, '
                                             'stdev=860.28\n'
                                             '    clat (usec): min=5, max=218, '
                                             'avg=33.43, stdev= 8.92\n'
                                             '     lat (usec): min=15, '
                                             'max=221, avg=35.45, stdev= 8.81\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894184, '
                                             'max=897432, per=100.00%, '
                                             'avg=895345.00, stdev=952.53, '
                                             'samples=14\n'
                                             '   iops        : min=223546, '
                                             'max=224358, avg=223836.21, '
                                             'stdev=238.15, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.23%, 50=92.13%, 100=7.36%, '
                                             '250=0.28%\n'
                                             '  cpu          : usr=16.41%, '
                                             'sys=62.79%, ctx=539835, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566663,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6120MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.output'),
                                   'state': {'begin': 1666727590.2101974,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7959024906158447,
                                             'end': 1666727591.0061,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_354.state')},
                       'cmd_355': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.output'),
                                   'state': {'begin': 1666727591.9957767,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7794620990753174,
                                             'end': 1666727592.7752388,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_355.state')},
                       'cmd_356': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.output'),
                                   'state': {'begin': 1666727592.7774534,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.05287218093872,
                                             'end': 1666727603.8303256,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_356.state')},
                       'cmd_357': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734693,\n'
                                             '  "timestamp_ms" : '
                                             '1666734693305,\n'
                                             '  "time" : "Tue Oct 25 23:51:33 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6416519168,\n'
                                             '        "io_kbytes" : 6266132,\n'
                                             '        "bw_bytes" : 916514664,\n'
                                             '        "bw" : 895033,\n'
                                             '        "iops" : 223757.320383,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566525,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1283,\n'
                                             '          "max" : 37260,\n'
                                             '          "mean" : 2104.457766,\n'
                                             '          "stddev" : '
                                             '885.717940,\n'
                                             '          "N" : 1566526\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 220,\n'
                                             '          "max" : 267692,\n'
                                             '          "mean" : '
                                             '33332.316245,\n'
                                             '          "stddev" : '
                                             '8926.514508,\n'
                                             '          "N" : 1566532,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24192,\n'
                                             '            "5.000000" : 28032,\n'
                                             '            "10.000000" : '
                                             '28032,\n'
                                             '            "20.000000" : '
                                             '28288,\n'
                                             '            "30.000000" : '
                                             '29568,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31872,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15189,\n'
                                             '          "max" : 269315,\n'
                                             '          "mean" : '
                                             '35436.773715,\n'
                                             '          "stddev" : '
                                             '8807.724468,\n'
                                             '          "N" : 1566532\n'
                                             '        },\n'
                                             '        "bw_min" : 892808,\n'
                                             '        "bw_max" : 897400,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895258.142857,\n'
                                             '        "bw_dev" : 1198.668098,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223202,\n'
                                             '        "iops_max" : 224350,\n'
                                             '        "iops_mean" : '
                                             '223814.500000,\n'
                                             '        "iops_stddev" : '
                                             '299.664107,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.414286,\n'
                                             '      "sys_cpu" : 64.100000,\n'
                                             '      "ctx" : 529590,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.010000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.256364,\n'
                                             '        "50" : 92.233191,\n'
                                             '        "100" : 7.231356,\n'
                                             '        "250" : 0.279089,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89557: Tue '
                                             'Oct 25 23:51:33 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6119MiB/7001msec)\n'
                                             '    slat (nsec): min=1283, '
                                             'max=37260, avg=2104.46, '
                                             'stdev=885.72\n'
                                             '    clat (nsec): min=220, '
                                             'max=267692, avg=33332.32, '
                                             'stdev=8926.51\n'
                                             '     lat (usec): min=15, '
                                             'max=269, avg=35.44, stdev= 8.81\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892808, '
                                             'max=897400, per=100.00%, '
                                             'avg=895258.14, stdev=1198.67, '
                                             'samples=14\n'
                                             '   iops        : min=223202, '
                                             'max=224350, avg=223814.50, '
                                             'stdev=299.66, samples=14\n'
                                             '  lat (nsec)   : 250=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.26%, 50=92.23%, '
                                             '100=7.23%\n'
                                             '  lat (usec)   : 250=0.28%, '
                                             '500=0.01%\n'
                                             '  cpu          : usr=15.41%, '
                                             'sys=64.10%, ctx=529590, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566525,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6119MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.output'),
                                   'state': {'begin': 1666727603.8315434,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.783583402633667,
                                             'end': 1666727604.6151268,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_357.state')},
                       'cmd_358': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_io_uring-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.output'),
                                   'state': {'begin': 1666727605.6035314,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7908978462219238,
                                             'end': 1666727606.3944292,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_358.state')},
                       'cmd_359': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.output'),
                                   'state': {'begin': 1666727606.3975043,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="io_uring-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="io_uring" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_io_uring-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.065941095352173,
                                             'end': 1666727617.4634454,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_359.state')},
                       'cmd_36': {'output': 'io_uring-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=io_uring, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733157,\n'
                                            '  "timestamp_ms" : '
                                            '1666733157028,\n'
                                            '  "time" : "Tue Oct 25 23:25:57 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "io_uring",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1804107776,\n'
                                            '        "io_kbytes" : 1761824,\n'
                                            '        "bw_bytes" : 257729682,\n'
                                            '        "bw" : 251689,\n'
                                            '        "iops" : 62922.285714,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 440456,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2465,\n'
                                            '          "max" : 34736,\n'
                                            '          "mean" : 2646.838867,\n'
                                            '          "stddev" : 213.387163,\n'
                                            '          "N" : 440456\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 240,\n'
                                            '          "max" : 105898,\n'
                                            '          "mean" : 12567.897549,\n'
                                            '          "stddev" : '
                                            '1992.233414,\n'
                                            '          "N" : 440456,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 12096,\n'
                                            '            "5.000000" : 12224,\n'
                                            '            "10.000000" : 12224,\n'
                                            '            "20.000000" : 12224,\n'
                                            '            "30.000000" : 12224,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12352,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12352,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22400,\n'
                                            '            "99.500000" : 27520,\n'
                                            '            "99.900000" : 36608,\n'
                                            '            "99.950000" : 49920,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 11952,\n'
                                            '          "max" : 108593,\n'
                                            '          "mean" : 15214.736416,\n'
                                            '          "stddev" : '
                                            '2006.232859,\n'
                                            '          "N" : 440456\n'
                                            '        },\n'
                                            '        "bw_min" : 250789,\n'
                                            '        "bw_max" : 252440,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '251748.785714,\n'
                                            '        "bw_dev" : 511.742450,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 62697,\n'
                                            '        "iops_max" : 63110,\n'
                                            '        "iops_mean" : '
                                            '62937.142857,\n'
                                            '        "iops_stddev" : '
                                            '127.995106,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.485714,\n'
                                            '      "sys_cpu" : 38.314286,\n'
                                            '      "ctx" : 440441,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.659571,\n'
                                            '        "50" : 1.284351,\n'
                                            '        "100" : 0.047905,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 623550,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5990,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5990,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=84848: Tue '
                                            'Oct 25 23:25:57 2022\n'
                                            '  read: IOPS=62.9k, BW=246MiB/s '
                                            '(258MB/s)(1721MiB/7000msec)\n'
                                            '    slat (nsec): min=2465, '
                                            'max=34736, avg=2646.84, '
                                            'stdev=213.39\n'
                                            '    clat (nsec): min=240, '
                                            'max=105898, avg=12567.90, '
                                            'stdev=1992.23\n'
                                            '     lat (usec): min=11, max=108, '
                                            'avg=15.21, stdev= 2.01\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[12096],  '
                                            '5.00th=[12224], 10.00th=[12224], '
                                            '20.00th=[12224],\n'
                                            '     | 30.00th=[12224], '
                                            '40.00th=[12224], 50.00th=[12352], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12352], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22400], '
                                            '99.50th=[27520], 99.90th=[36608], '
                                            '99.95th=[49920],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=250789, '
                                            'max=252440, per=100.00%, '
                                            'avg=251748.79, stdev=511.74, '
                                            'samples=14\n'
                                            '   iops        : min=62697, '
                                            'max=63110, avg=62937.14, '
                                            'stdev=128.00, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.66%, 50=1.28%, '
                                            '100=0.05%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=9.49%, '
                                            'sys=38.31%, ctx=440441, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=440456,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=246MiB/s (258MB/s), '
                                            '246MiB/s-246MiB/s '
                                            '(258MB/s-258MB/s), io=1721MiB '
                                            '(1804MB), run=7000-7000msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=623550/0, '
                                            'merge=0/0, ticks=5990/0, '
                                            'in_queue=5990, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.output'),
                                  'state': {'begin': 1666726067.5289435,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7989840507507324,
                                            'end': 1666726068.3279276,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_36.state')},
                       'cmd_360': {'output': 'io_uring-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734706,\n'
                                             '  "timestamp_ms" : '
                                             '1666734706937,\n'
                                             '  "time" : "Tue Oct 25 23:51:46 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"io_uring-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"io_uring-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"io_uring",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6417715200,\n'
                                             '        "io_kbytes" : 6267300,\n'
                                             '        "bw_bytes" : 916685502,\n'
                                             '        "bw" : 895200,\n'
                                             '        "iops" : 223799.028710,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566817,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1292,\n'
                                             '          "max" : 57117,\n'
                                             '          "mean" : 2005.485569,\n'
                                             '          "stddev" : '
                                             '861.647587,\n'
                                             '          "N" : 1566818\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 261,\n'
                                             '          "max" : 206247,\n'
                                             '          "mean" : '
                                             '33462.330400,\n'
                                             '          "stddev" : '
                                             '8878.873955,\n'
                                             '          "N" : 1566824,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23936,\n'
                                             '            "5.000000" : 28032,\n'
                                             '            "10.000000" : '
                                             '28032,\n'
                                             '            "20.000000" : '
                                             '28544,\n'
                                             '            "30.000000" : '
                                             '30336,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31872,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '32640,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 16311,\n'
                                             '          "max" : 207599,\n'
                                             '          "mean" : '
                                             '35467.815561,\n'
                                             '          "stddev" : '
                                             '8771.771322,\n'
                                             '          "N" : 1566824\n'
                                             '        },\n'
                                             '        "bw_min" : 893368,\n'
                                             '        "bw_max" : 897939,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895407.071429,\n'
                                             '        "bw_dev" : 1490.748416,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223342,\n'
                                             '        "iops_max" : 224484,\n'
                                             '        "iops_mean" : '
                                             '223851.714286,\n'
                                             '        "iops_stddev" : '
                                             '372.323149,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 15.357143,\n'
                                             '      "sys_cpu" : 63.742857,\n'
                                             '      "ctx" : 541721,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.272080,\n'
                                             '        "50" : 92.033020,\n'
                                             '        "100" : 7.417778,\n'
                                             '        "250" : 0.277314,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'io_uring-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89597: Tue '
                                             'Oct 25 23:51:46 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6120MiB/7001msec)\n'
                                             '    slat (nsec): min=1292, '
                                             'max=57117, avg=2005.49, '
                                             'stdev=861.65\n'
                                             '    clat (nsec): min=261, '
                                             'max=206247, avg=33462.33, '
                                             'stdev=8878.87\n'
                                             '     lat (usec): min=16, '
                                             'max=207, avg=35.47, stdev= 8.77\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   33], 90.00th=[   '
                                             '46], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893368, '
                                             'max=897939, per=100.00%, '
                                             'avg=895407.07, stdev=1490.75, '
                                             'samples=14\n'
                                             '   iops        : min=223342, '
                                             'max=224484, avg=223851.71, '
                                             'stdev=372.32, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.27%, 50=92.03%, '
                                             '100=7.42%\n'
                                             '  lat (usec)   : 250=0.28%\n'
                                             '  cpu          : usr=15.36%, '
                                             'sys=63.74%, ctx=541721, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566817,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6120MiB '
                                             '(6418MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.output'),
                                   'state': {'begin': 1666727617.4662776,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_io_uring-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7870941162109375,
                                             'end': 1666727618.2533717,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_360.state')},
                       'cmd_361': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-reference_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.output'),
                                   'state': {'begin': 1666727619.2475295,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7842357158660889,
                                             'end': 1666727620.0317652,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_361.state')},
                       'cmd_362': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.output'),
                                   'state': {'begin': 1666727620.033275,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-reference_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.068465232849121,
                                             'end': 1666727631.1017401,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_362.state')},
                       'cmd_363': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734720,\n'
                                             '  "timestamp_ms" : '
                                             '1666734720577,\n'
                                             '  "time" : "Tue Oct 25 23:52:00 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6421184512,\n'
                                             '        "io_kbytes" : 6270688,\n'
                                             '        "bw_bytes" : 917181047,\n'
                                             '        "bw" : 895684,\n'
                                             '        "iops" : 223920.011427,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567664,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1372,\n'
                                             '          "max" : 23945,\n'
                                             '          "mean" : 2044.074586,\n'
                                             '          "stddev" : '
                                             '778.052273,\n'
                                             '          "N" : 1567665\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 15299,\n'
                                             '          "max" : 193182,\n'
                                             '          "mean" : '
                                             '33405.325188,\n'
                                             '          "stddev" : '
                                             '8836.893542,\n'
                                             '          "N" : 1567671,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 28288,\n'
                                             '            "10.000000" : '
                                             '28288,\n'
                                             '            "20.000000" : '
                                             '28800,\n'
                                             '            "30.000000" : '
                                             '29312,\n'
                                             '            "40.000000" : '
                                             '30592,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '128512,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 17783,\n'
                                             '          "max" : 194615,\n'
                                             '          "mean" : '
                                             '35449.399103,\n'
                                             '          "stddev" : '
                                             '8754.063693,\n'
                                             '          "N" : 1567671\n'
                                             '        },\n'
                                             '        "bw_min" : 894608,\n'
                                             '        "bw_max" : 897456,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '896045.714286,\n'
                                             '        "bw_dev" : 800.433289,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223652,\n'
                                             '        "iops_max" : 224364,\n'
                                             '        "iops_mean" : '
                                             '224011.428571,\n'
                                             '        "iops_stddev" : '
                                             '200.229758,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 21.657143,\n'
                                             '      "sys_cpu" : 57.442857,\n'
                                             '      "ctx" : 533446,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.044142,\n'
                                             '        "50" : 92.606005,\n'
                                             '        "100" : 7.090231,\n'
                                             '        "250" : 0.260068,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2235295,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 71869,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 71869,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89636: Tue '
                                             'Oct 25 23:52:00 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6124MiB/7001msec)\n'
                                             '    slat (nsec): min=1372, '
                                             'max=23945, avg=2044.07, '
                                             'stdev=778.05\n'
                                             '    clat (usec): min=15, '
                                             'max=193, avg=33.41, stdev= 8.84\n'
                                             '     lat (usec): min=17, '
                                             'max=194, avg=35.45, stdev= 8.75\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '45], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '129], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894608, '
                                             'max=897456, per=100.00%, '
                                             'avg=896045.71, stdev=800.43, '
                                             'samples=14\n'
                                             '   iops        : min=223652, '
                                             'max=224364, avg=224011.43, '
                                             'stdev=200.23, samples=14\n'
                                             '  lat (usec)   : 20=0.04%, '
                                             '50=92.61%, 100=7.09%, 250=0.26%\n'
                                             '  cpu          : usr=21.66%, '
                                             'sys=57.44%, ctx=533446, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567664,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6124MiB '
                                             '(6421MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2235295/0, '
                                             'merge=0/0, ticks=71869/0, '
                                             'in_queue=71869, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.output'),
                                   'state': {'begin': 1666727631.1085894,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-reference_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.780418872833252,
                                             'end': 1666727631.8890083,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_363.state')},
                       'cmd_364': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-reference_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.output'),
                                   'state': {'begin': 1666727632.8713968,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7804844379425049,
                                             'end': 1666727633.6518812,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_364.state')},
                       'cmd_365': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.output'),
                                   'state': {'begin': 1666727633.653573,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-reference_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.066412925720215,
                                             'end': 1666727644.719986,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_365.state')},
                       'cmd_366': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734734,\n'
                                             '  "timestamp_ms" : '
                                             '1666734734194,\n'
                                             '  "time" : "Tue Oct 25 23:52:14 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419886080,\n'
                                             '        "io_kbytes" : 6269420,\n'
                                             '        "bw_bytes" : 916995583,\n'
                                             '        "bw" : 895503,\n'
                                             '        "iops" : 223874.875018,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567348,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1422,\n'
                                             '          "max" : 42911,\n'
                                             '          "mean" : 2070.390944,\n'
                                             '          "stddev" : '
                                             '758.523721,\n'
                                             '          "N" : 1567348\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 391,\n'
                                             '          "max" : 177383,\n'
                                             '          "mean" : '
                                             '33384.698160,\n'
                                             '          "stddev" : '
                                             '8925.881402,\n'
                                             '          "N" : 1567355,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 22912,\n'
                                             '            "5.000000" : 27264,\n'
                                             '            "10.000000" : '
                                             '28544,\n'
                                             '            "20.000000" : '
                                             '28800,\n'
                                             '            "30.000000" : '
                                             '29056,\n'
                                             '            "40.000000" : '
                                             '29568,\n'
                                             '            "50.000000" : '
                                             '30848,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32640,\n'
                                             '            "80.000000" : '
                                             '34048,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '128512,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14407,\n'
                                             '          "max" : 178875,\n'
                                             '          "mean" : '
                                             '35455.088327,\n'
                                             '          "stddev" : '
                                             '8862.894253,\n'
                                             '          "N" : 1567355\n'
                                             '        },\n'
                                             '        "bw_min" : 893552,\n'
                                             '        "bw_max" : 897856,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895919.923077,\n'
                                             '        "bw_dev" : 1175.135344,\n'
                                             '        "bw_samples" : 13,\n'
                                             '        "iops_min" : 223388,\n'
                                             '        "iops_max" : 224464,\n'
                                             '        "iops_mean" : '
                                             '223979.923077,\n'
                                             '        "iops_stddev" : '
                                             '293.956477,\n'
                                             '        "iops_samples" : 13\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 25.357143,\n'
                                             '      "sys_cpu" : 52.514286,\n'
                                             '      "ctx" : 563635,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.058570,\n'
                                             '        "50" : 92.628185,\n'
                                             '        "100" : 7.051848,\n'
                                             '        "250" : 0.260950,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2177732,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 69367,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 69367,\n'
                                             '      "util" : 99.085923\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89676: Tue '
                                             'Oct 25 23:52:14 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=1422, '
                                             'max=42911, avg=2070.39, '
                                             'stdev=758.52\n'
                                             '    clat (nsec): min=391, '
                                             'max=177383, avg=33384.70, '
                                             'stdev=8925.88\n'
                                             '     lat (usec): min=14, '
                                             'max=178, avg=35.46, stdev= 8.86\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   23],  '
                                             '5.00th=[   28], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   30], 50.00th=[   '
                                             '31], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   35], 90.00th=[   '
                                             '45], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '129], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893552, '
                                             'max=897856, per=100.00%, '
                                             'avg=895919.92, stdev=1175.14, '
                                             'samples=13\n'
                                             '   iops        : min=223388, '
                                             'max=224464, avg=223979.92, '
                                             'stdev=293.96, samples=13\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '20=0.06%, 50=92.63%, 100=7.05%, '
                                             '250=0.26%\n'
                                             '  cpu          : usr=25.36%, '
                                             'sys=52.51%, ctx=563635, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567348,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6420MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2177732/0, '
                                             'merge=0/0, ticks=69367/0, '
                                             'in_queue=69367, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.output'),
                                   'state': {'begin': 1666727644.7220576,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-reference_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.79469895362854,
                                             'end': 1666727645.5167565,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_366.state')},
                       'cmd_367': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-reference_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.output'),
                                   'state': {'begin': 1666727646.5035763,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7806024551391602,
                                             'end': 1666727647.2841787,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_367.state')},
                       'cmd_368': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.output'),
                                   'state': {'begin': 1666727647.285832,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-reference" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="libaio" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-reference_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.056997060775757,
                                             'end': 1666727658.342829,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_368.state')},
                       'cmd_369': {'output': 'libaio-reference: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=libaio, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734747,\n'
                                             '  "timestamp_ms" : '
                                             '1666734747818,\n'
                                             '  "time" : "Tue Oct 25 23:52:27 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-reference",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-reference",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : "libaio",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419091456,\n'
                                             '        "io_kbytes" : 6268644,\n'
                                             '        "bw_bytes" : 916882081,\n'
                                             '        "bw" : 895392,\n'
                                             '        "iops" : 223847.164691,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567154,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1332,\n'
                                             '          "max" : 1297263,\n'
                                             '          "mean" : 2019.222772,\n'
                                             '          "stddev" : '
                                             '1292.972535,\n'
                                             '          "N" : 1567154\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 12954,\n'
                                             '          "max" : 1329383,\n'
                                             '          "mean" : '
                                             '33446.079906,\n'
                                             '          "stddev" : '
                                             '9235.556673,\n'
                                             '          "N" : 1567161,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24192,\n'
                                             '            "5.000000" : 28032,\n'
                                             '            "10.000000" : '
                                             '28288,\n'
                                             '            "20.000000" : '
                                             '28800,\n'
                                             '            "30.000000" : '
                                             '29568,\n'
                                             '            "40.000000" : '
                                             '30592,\n'
                                             '            "50.000000" : '
                                             '31616,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '33536,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 17462,\n'
                                             '          "max" : 1332139,\n'
                                             '          "mean" : '
                                             '35465.301490,\n'
                                             '          "stddev" : '
                                             '9217.281644,\n'
                                             '          "N" : 1567161\n'
                                             '        },\n'
                                             '        "bw_min" : 892584,\n'
                                             '        "bw_max" : 897699,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895743.142857,\n'
                                             '        "bw_dev" : 1372.428272,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223146,\n'
                                             '        "iops_max" : 224424,\n'
                                             '        "iops_mean" : '
                                             '223935.714286,\n'
                                             '        "iops_stddev" : '
                                             '343.066584,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.885714,\n'
                                             '      "sys_cpu" : 55.914286,\n'
                                             '      "ctx" : 540455,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999936,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.049261,\n'
                                             '        "50" : 92.557081,\n'
                                             '        "100" : 7.134079,\n'
                                             '        "250" : 0.259579,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ],\n'
                                             '  "disk_util" : [\n'
                                             '    {\n'
                                             '      "name" : "nvme0n1",\n'
                                             '      "read_ios" : 2234710,\n'
                                             '      "write_ios" : 0,\n'
                                             '      "read_merges" : 0,\n'
                                             '      "write_merges" : 0,\n'
                                             '      "read_ticks" : 71868,\n'
                                             '      "write_ticks" : 0,\n'
                                             '      "in_queue" : 71868,\n'
                                             '      "util" : 99.088748\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-reference: (groupid=0, '
                                             'jobs=1): err= 0: pid=89716: Tue '
                                             'Oct 25 23:52:27 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=1332, '
                                             'max=1297.3k, avg=2019.22, '
                                             'stdev=1292.97\n'
                                             '    clat (usec): min=12, '
                                             'max=1329, avg=33.45, stdev= '
                                             '9.24\n'
                                             '     lat (usec): min=17, '
                                             'max=1332, avg=35.47, stdev= '
                                             '9.22\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   29], 10.00th=[   29], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   34], 90.00th=[   '
                                             '45], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892584, '
                                             'max=897699, per=100.00%, '
                                             'avg=895743.14, stdev=1372.43, '
                                             'samples=14\n'
                                             '   iops        : min=223146, '
                                             'max=224424, avg=223935.71, '
                                             'stdev=343.07, samples=14\n'
                                             '  lat (usec)   : 20=0.05%, '
                                             '50=92.56%, 100=7.13%, 250=0.26%\n'
                                             '  lat (msec)   : 2=0.01%\n'
                                             '  cpu          : usr=22.89%, '
                                             'sys=55.91%, ctx=540455, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567154,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6419MB), run=7001-7001msec\n'
                                             '\n'
                                             'Disk stats (read/write):\n'
                                             '  nvme0n1: ios=2234710/0, '
                                             'merge=0/0, ticks=71868/0, '
                                             'in_queue=71868, util=99.09%\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.output'),
                                   'state': {'begin': 1666727658.3446329,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-reference_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7802093029022217,
                                             'end': 1666727659.1248422,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_369.state')},
                       'cmd_37': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.output'),
                                  'state': {'begin': 1666726070.3615396,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7853639125823975,
                                            'end': 1666726071.1469035,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_37.state')},
                       'cmd_370': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.output'),
                                   'state': {'begin': 1666727661.132002,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7778584957122803,
                                             'end': 1666727661.9098606,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_370.state')},
                       'cmd_371': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.output'),
                                   'state': {'begin': 1666727661.9113603,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.515507459640503,
                                             'end': 1666727673.4268677,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_371.state')},
                       'cmd_372': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734762,\n'
                                             '  "timestamp_ms" : '
                                             '1666734762792,\n'
                                             '  "time" : "Tue Oct 25 23:52:42 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6415536128,\n'
                                             '        "io_kbytes" : 6265172,\n'
                                             '        "bw_bytes" : 916374250,\n'
                                             '        "bw" : 894896,\n'
                                             '        "iops" : 223723.039566,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566285,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1553,\n'
                                             '          "max" : 184837,\n'
                                             '          "mean" : 1667.843817,\n'
                                             '          "stddev" : '
                                             '320.534687,\n'
                                             '          "N" : 1566286\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 541,\n'
                                             '          "max" : 215374,\n'
                                             '          "mean" : '
                                             '33789.432111,\n'
                                             '          "stddev" : '
                                             '8671.015722,\n'
                                             '          "N" : 1566285,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31360,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '134144,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14217,\n'
                                             '          "max" : 217007,\n'
                                             '          "mean" : '
                                             '35457.275931,\n'
                                             '          "stddev" : '
                                             '8658.022173,\n'
                                             '          "N" : 1566285\n'
                                             '        },\n'
                                             '        "bw_min" : 893960,\n'
                                             '        "bw_max" : 896200,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895105.071429,\n'
                                             '        "bw_dev" : 806.860627,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223490,\n'
                                             '        "iops_max" : 224050,\n'
                                             '        "iops_mean" : '
                                             '223776.214286,\n'
                                             '        "iops_stddev" : '
                                             '201.682225,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 67.585714,\n'
                                             '      "sys_cpu" : 32.385714,\n'
                                             '      "ctx" : 14,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.306531,\n'
                                             '        "8" : 98.689702,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.996755,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.010000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.047437,\n'
                                             '        "50" : 90.986251,\n'
                                             '        "100" : 8.674794,\n'
                                             '        "250" : 0.291326,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=89765: Tue '
                                             'Oct 25 23:52:42 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6118MiB/7001msec)\n'
                                             '    slat (nsec): min=1553, '
                                             'max=184837, avg=1667.84, '
                                             'stdev=320.53\n'
                                             '    clat (nsec): min=541, '
                                             'max=215374, avg=33789.43, '
                                             'stdev=8671.02\n'
                                             '     lat (usec): min=14, '
                                             'max=217, avg=35.46, stdev= 8.66\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '135], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893960, '
                                             'max=896200, per=100.00%, '
                                             'avg=895105.07, stdev=806.86, '
                                             'samples=14\n'
                                             '   iops        : min=223490, '
                                             'max=224050, avg=223776.21, '
                                             'stdev=201.68, samples=14\n'
                                             '  lat (nsec)   : 750=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.05%, 50=90.99%, '
                                             '100=8.67%\n'
                                             '  lat (usec)   : 250=0.29%\n'
                                             '  cpu          : usr=67.59%, '
                                             'sys=32.39%, ctx=14, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.3%, 8=98.7%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566285,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6118MiB '
                                             '(6416MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.output'),
                                   'state': {'begin': 1666727673.4297333,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7888956069946289,
                                             'end': 1666727674.218629,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_372.state')},
                       'cmd_373': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.output'),
                                   'state': {'begin': 1666727676.2286882,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7789840698242188,
                                             'end': 1666727677.0076723,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_373.state')},
                       'cmd_374': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.output'),
                                   'state': {'begin': 1666727677.0096018,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.522533655166626,
                                             'end': 1666727688.5321355,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_374.state')},
                       'cmd_375': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734777,\n'
                                             '  "timestamp_ms" : '
                                             '1666734777907,\n'
                                             '  "time" : "Tue Oct 25 23:52:57 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419636224,\n'
                                             '        "io_kbytes" : 6269176,\n'
                                             '        "bw_bytes" : 916959894,\n'
                                             '        "bw" : 895468,\n'
                                             '        "iops" : 223866.161977,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567287,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1562,\n'
                                             '          "max" : 23695,\n'
                                             '          "mean" : 1655.664724,\n'
                                             '          "stddev" : '
                                             '232.107596,\n'
                                             '          "N" : 1567287\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 7424,\n'
                                             '          "max" : 174908,\n'
                                             '          "mean" : '
                                             '33773.507443,\n'
                                             '          "stddev" : '
                                             '8548.878283,\n'
                                             '          "N" : 1567294,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '31104,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31360,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 14487,\n'
                                             '          "max" : 176551,\n'
                                             '          "mean" : '
                                             '35429.171972,\n'
                                             '          "stddev" : '
                                             '8534.978594,\n'
                                             '          "N" : 1567294\n'
                                             '        },\n'
                                             '        "bw_min" : 892608,\n'
                                             '        "bw_max" : 898528,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895684.785714,\n'
                                             '        "bw_dev" : 1449.683476,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223152,\n'
                                             '        "iops_max" : 224632,\n'
                                             '        "iops_mean" : '
                                             '223921.142857,\n'
                                             '        "iops_stddev" : '
                                             '362.336617,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 68.328571,\n'
                                             '      "sys_cpu" : 31.642857,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.275835,\n'
                                             '        "8" : 98.722952,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.997849,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.037517,\n'
                                             '        "50" : 91.020981,\n'
                                             '        "100" : 8.667525,\n'
                                             '        "250" : 0.274359,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=89814: Tue '
                                             'Oct 25 23:52:57 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=1562, '
                                             'max=23695, avg=1655.66, '
                                             'stdev=232.11\n'
                                             '    clat (usec): min=7, max=174, '
                                             'avg=33.77, stdev= 8.55\n'
                                             '     lat (usec): min=14, '
                                             'max=176, avg=35.43, stdev= 8.53\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   32], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '131], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=892608, '
                                             'max=898528, per=100.00%, '
                                             'avg=895684.79, stdev=1449.68, '
                                             'samples=14\n'
                                             '   iops        : min=223152, '
                                             'max=224632, avg=223921.14, '
                                             'stdev=362.34, samples=14\n'
                                             '  lat (usec)   : 10=0.01%, '
                                             '20=0.04%, 50=91.02%, 100=8.67%, '
                                             '250=0.27%\n'
                                             '  cpu          : usr=68.33%, '
                                             'sys=31.64%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.3%, 8=98.7%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567287,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6420MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.output'),
                                   'state': {'begin': 1666727688.5368567,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7866065502166748,
                                             'end': 1666727689.3234632,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_375.state')},
                       'cmd_376': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.output'),
                                   'state': {'begin': 1666727691.3690376,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7804043292999268,
                                             'end': 1666727692.149442,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_376.state')},
                       'cmd_377': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.output'),
                                   'state': {'begin': 1666727692.1515708,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_aio" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.519450426101685,
                                             'end': 1666727703.6710212,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_377.state')},
                       'cmd_378': {'output': 'libaio-bdev_aio: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734793,\n'
                                             '  "timestamp_ms" : '
                                             '1666734793038,\n'
                                             '  "time" : "Tue Oct 25 23:53:13 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_aio",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_aio",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6418944000,\n'
                                             '        "io_kbytes" : 6268500,\n'
                                             '        "bw_bytes" : 916861019,\n'
                                             '        "bw" : 895372,\n'
                                             '        "iops" : 223841.879731,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567117,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1542,\n'
                                             '          "max" : 51917,\n'
                                             '          "mean" : 1638.791215,\n'
                                             '          "stddev" : '
                                             '252.351930,\n'
                                             '          "N" : 1567118\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 12954,\n'
                                             '          "max" : 173766,\n'
                                             '          "mean" : '
                                             '33817.441936,\n'
                                             '          "stddev" : '
                                             '8542.652012,\n'
                                             '          "N" : 1567117,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 30848,\n'
                                             '            "10.000000" : '
                                             '31104,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31360,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31616,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '66048,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15129,\n'
                                             '          "max" : 175399,\n'
                                             '          "mean" : '
                                             '35456.233161,\n'
                                             '          "stddev" : '
                                             '8527.216217,\n'
                                             '          "N" : 1567117\n'
                                             '        },\n'
                                             '        "bw_min" : 894312,\n'
                                             '        "bw_max" : 896672,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895574.142857,\n'
                                             '        "bw_dev" : 761.594467,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223578,\n'
                                             '        "iops_max" : 224168,\n'
                                             '        "iops_mean" : '
                                             '223893.500000,\n'
                                             '        "iops_stddev" : '
                                             '190.409579,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 71.014286,\n'
                                             '      "sys_cpu" : 28.971429,\n'
                                             '      "ctx" : 46,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.324087,\n'
                                             '        "8" : 98.671446,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.995953,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.057941,\n'
                                             '        "50" : 90.970872,\n'
                                             '        "100" : 8.694820,\n'
                                             '        "250" : 0.276367,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_aio: (groupid=0, '
                                             'jobs=1): err= 0: pid=89863: Tue '
                                             'Oct 25 23:53:13 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6122MiB/7001msec)\n'
                                             '    slat (nsec): min=1542, '
                                             'max=51917, avg=1638.79, '
                                             'stdev=252.35\n'
                                             '    clat (usec): min=12, '
                                             'max=173, avg=33.82, stdev= 8.54\n'
                                             '     lat (usec): min=15, '
                                             'max=175, avg=35.46, stdev= 8.53\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   32], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   67], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894312, '
                                             'max=896672, per=100.00%, '
                                             'avg=895574.14, stdev=761.59, '
                                             'samples=14\n'
                                             '   iops        : min=223578, '
                                             'max=224168, avg=223893.50, '
                                             'stdev=190.41, samples=14\n'
                                             '  lat (usec)   : 20=0.06%, '
                                             '50=90.97%, 100=8.69%, 250=0.28%\n'
                                             '  cpu          : usr=71.01%, '
                                             'sys=28.97%, ctx=46, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.3%, 8=98.7%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567117,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6122MiB '
                                             '(6419MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.output'),
                                   'state': {'begin': 1666727703.6730046,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7829241752624512,
                                             'end': 1666727704.4559288,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_378.state')},
                       'cmd_379': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.output'),
                                   'state': {'begin': 1666727706.523488,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7925660610198975,
                                             'end': 1666727707.316054,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_379.state')},
                       'cmd_38': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.output'),
                                  'state': {'begin': 1666726071.149084,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.456365823745728,
                                            'end': 1666726082.60545,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_38.state')},
                       'cmd_380': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.output'),
                                   'state': {'begin': 1666727707.3180034,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.541755437850952,
                                             'end': 1666727718.8597589,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_380.state')},
                       'cmd_381': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734808,\n'
                                             '  "timestamp_ms" : '
                                             '1666734808232,\n'
                                             '  "time" : "Tue Oct 25 23:53:28 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6424309760,\n'
                                             '        "io_kbytes" : 6273740,\n'
                                             '        "bw_bytes" : 917627447,\n'
                                             '        "bw" : 896120,\n'
                                             '        "iops" : 224028.995858,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1568427,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1683,\n'
                                             '          "max" : 37610,\n'
                                             '          "mean" : 1920.989116,\n'
                                             '          "stddev" : '
                                             '147.325480,\n'
                                             '          "N" : 1568428\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1113,\n'
                                             '          "max" : 185858,\n'
                                             '          "mean" : '
                                             '33505.838697,\n'
                                             '          "stddev" : '
                                             '8743.561843,\n'
                                             '          "N" : 1568430,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23168,\n'
                                             '            "5.000000" : 28288,\n'
                                             '            "10.000000" : '
                                             '30336,\n'
                                             '            "20.000000" : '
                                             '30592,\n'
                                             '            "30.000000" : '
                                             '30848,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '30848,\n'
                                             '            "60.000000" : '
                                             '31104,\n'
                                             '            "70.000000" : '
                                             '31104,\n'
                                             '            "80.000000" : '
                                             '31360,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '142336,\n'
                                             '            "99.990000" : '
                                             '160768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13776,\n'
                                             '          "max" : 187792,\n'
                                             '          "mean" : '
                                             '35426.827932,\n'
                                             '          "stddev" : '
                                             '8743.318794,\n'
                                             '          "N" : 1568430\n'
                                             '        },\n'
                                             '        "bw_min" : 894136,\n'
                                             '        "bw_max" : 898096,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '896613.785714,\n'
                                             '        "bw_dev" : 1166.083001,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223534,\n'
                                             '        "iops_max" : 224524,\n'
                                             '        "iops_mean" : '
                                             '224153.357143,\n'
                                             '        "iops_stddev" : '
                                             '291.452492,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 41.528571,\n'
                                             '      "sys_cpu" : 58.442857,\n'
                                             '      "ctx" : 18,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 0.739403,\n'
                                             '        "8" : 99.259832,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999232,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.127963,\n'
                                             '        "50" : 91.673441,\n'
                                             '        "100" : 7.954977,\n'
                                             '        "250" : 0.243747,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89912: Tue '
                                             'Oct 25 23:53:28 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(918MB/s)(6127MiB/7001msec)\n'
                                             '    slat (nsec): min=1683, '
                                             'max=37610, avg=1920.99, '
                                             'stdev=147.33\n'
                                             '    clat (nsec): min=1113, '
                                             'max=185858, avg=33505.84, '
                                             'stdev=8743.56\n'
                                             '     lat (usec): min=13, '
                                             'max=187, avg=35.43, stdev= 8.74\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   29], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '31], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '45], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '131], 99.95th=[  143],\n'
                                             '     | 99.99th=[  161]\n'
                                             '   bw (  KiB/s): min=894136, '
                                             'max=898096, per=100.00%, '
                                             'avg=896613.79, stdev=1166.08, '
                                             'samples=14\n'
                                             '   iops        : min=223534, '
                                             'max=224524, avg=224153.36, '
                                             'stdev=291.45, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '20=0.13%, 50=91.67%, 100=7.95%, '
                                             '250=0.24%\n'
                                             '  cpu          : usr=41.53%, '
                                             'sys=58.44%, ctx=18, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=0.7%, 8=99.3%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1568427,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (918MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(918MB/s-918MB/s), io=6127MiB '
                                             '(6424MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.output'),
                                   'state': {'begin': 1666727718.8625243,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7950255870819092,
                                             'end': 1666727719.6575499,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_381.state')},
                       'cmd_382': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.output'),
                                   'state': {'begin': 1666727721.717935,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7921743392944336,
                                             'end': 1666727722.5101094,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_382.state')},
                       'cmd_383': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.output'),
                                   'state': {'begin': 1666727722.5123904,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.539591789245605,
                                             'end': 1666727734.0519822,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_383.state')},
                       'cmd_384': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734823,\n'
                                             '  "timestamp_ms" : '
                                             '1666734823417,\n'
                                             '  "time" : "Tue Oct 25 23:53:43 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6421917696,\n'
                                             '        "io_kbytes" : 6271404,\n'
                                             '        "bw_bytes" : 917285772,\n'
                                             '        "bw" : 895786,\n'
                                             '        "iops" : 223945.722040,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567844,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1623,\n'
                                             '          "max" : 74419,\n'
                                             '          "mean" : 1755.833047,\n'
                                             '          "stddev" : '
                                             '331.760129,\n'
                                             '          "N" : 1567844\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1042,\n'
                                             '          "max" : 177283,\n'
                                             '          "mean" : '
                                             '33637.009474,\n'
                                             '          "stddev" : '
                                             '8459.789321,\n'
                                             '          "N" : 1567851,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24704,\n'
                                             '            "5.000000" : 30592,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '30848,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31360,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31616,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '64768,\n'
                                             '            "99.900000" : '
                                             '127488,\n'
                                             '            "99.950000" : '
                                             '138240,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 13115,\n'
                                             '          "max" : 178976,\n'
                                             '          "mean" : '
                                             '35392.842113,\n'
                                             '          "stddev" : '
                                             '8437.240273,\n'
                                             '          "N" : 1567851\n'
                                             '        },\n'
                                             '        "bw_min" : 894312,\n'
                                             '        "bw_max" : 897768,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '896021.142857,\n'
                                             '        "bw_dev" : 1070.104442,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223578,\n'
                                             '        "iops_max" : 224442,\n'
                                             '        "iops_mean" : '
                                             '224005.285714,\n'
                                             '        "iops_stddev" : '
                                             '267.558312,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 48.500000,\n'
                                             '      "sys_cpu" : 51.471429,\n'
                                             '      "ctx" : 25,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.606537,\n'
                                             '        "8" : 98.385299,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.995820,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.055618,\n'
                                             '        "50" : 91.209266,\n'
                                             '        "100" : 8.477757,\n'
                                             '        "250" : 0.257360,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=89963: Tue '
                                             'Oct 25 23:53:43 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6124MiB/7001msec)\n'
                                             '    slat (nsec): min=1623, '
                                             'max=74419, avg=1755.83, '
                                             'stdev=331.76\n'
                                             '    clat (nsec): min=1042, '
                                             'max=177283, avg=33637.01, '
                                             'stdev=8459.79\n'
                                             '     lat (usec): min=13, '
                                             'max=178, avg=35.39, stdev= 8.44\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   31],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   65], 99.90th=[  '
                                             '128], 99.95th=[  139],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894312, '
                                             'max=897768, per=100.00%, '
                                             'avg=896021.14, stdev=1070.10, '
                                             'samples=14\n'
                                             '   iops        : min=223578, '
                                             'max=224442, avg=224005.29, '
                                             'stdev=267.56, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.01%, 20=0.06%, 50=91.21%, '
                                             '100=8.48%\n'
                                             '  lat (usec)   : 250=0.26%\n'
                                             '  cpu          : usr=48.50%, '
                                             'sys=51.47%, ctx=25, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.6%, 8=98.4%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567844,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6124MiB '
                                             '(6422MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.output'),
                                   'state': {'begin': 1666727734.0538235,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7862935066223145,
                                             'end': 1666727734.840117,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_384.state')},
                       'cmd_385': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.output'),
                                   'state': {'begin': 1666727736.853087,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7882089614868164,
                                             'end': 1666727737.641296,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_385.state')},
                       'cmd_386': {'output': 'TELEMETRY: No legacy callbacks, '
                                             'legacy socket not created\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.output'),
                                   'state': {'begin': 1666727737.6432695,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-bdev_xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="spdk_bdev" '
                                                    '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                    '--filename="Nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.533258438110352,
                                             'end': 1666727749.176528,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_386.state')},
                       'cmd_387': {'output': 'libaio-bdev_xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=spdk_bdev, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734838,\n'
                                             '  "timestamp_ms" : '
                                             '1666734838539,\n'
                                             '  "time" : "Tue Oct 25 23:53:58 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-bdev_xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"spdk_bdev",\n'
                                             '        "spdk_conf" : '
                                             '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                             '        "filename" : "Nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6419918848,\n'
                                             '        "io_kbytes" : 6269452,\n'
                                             '        "bw_bytes" : 917000263,\n'
                                             '        "bw" : 895508,\n'
                                             '        "iops" : 223876.017712,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1567356,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1583,\n'
                                             '          "max" : 180989,\n'
                                             '          "mean" : 1700.893440,\n'
                                             '          "stddev" : '
                                             '355.436322,\n'
                                             '          "N" : 1567356\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 1022,\n'
                                             '          "max" : 221516,\n'
                                             '          "mean" : '
                                             '33718.714563,\n'
                                             '          "stddev" : '
                                             '8534.695940,\n'
                                             '          "N" : 1567363,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 24448,\n'
                                             '            "5.000000" : 30592,\n'
                                             '            "10.000000" : '
                                             '30848,\n'
                                             '            "20.000000" : '
                                             '31104,\n'
                                             '            "30.000000" : '
                                             '31104,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31360,\n'
                                             '            "70.000000" : '
                                             '31360,\n'
                                             '            "80.000000" : '
                                             '31616,\n'
                                             '            "90.000000" : '
                                             '43776,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '65280,\n'
                                             '            "99.900000" : '
                                             '129536,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15730,\n'
                                             '          "max" : 223139,\n'
                                             '          "mean" : '
                                             '35419.607731,\n'
                                             '          "stddev" : '
                                             '8515.962916,\n'
                                             '          "N" : 1567363\n'
                                             '        },\n'
                                             '        "bw_min" : 894160,\n'
                                             '        "bw_max" : 898496,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895716.928571,\n'
                                             '        "bw_dev" : 996.908025,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223540,\n'
                                             '        "iops_max" : 224624,\n'
                                             '        "iops_mean" : '
                                             '223929.357143,\n'
                                             '        "iops_stddev" : '
                                             '249.118324,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 48.785714,\n'
                                             '      "sys_cpu" : 51.185714,\n'
                                             '      "ctx" : 10,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.100000,\n'
                                             '        "2" : 0.100000,\n'
                                             '        "4" : 1.516120,\n'
                                             '        "8" : 98.479159,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.995950,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.010000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.054040,\n'
                                             '        "50" : 91.150256,\n'
                                             '        "100" : 8.527929,\n'
                                             '        "250" : 0.267967,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-bdev_xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=90012: Tue '
                                             'Oct 25 23:53:58 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6123MiB/7001msec)\n'
                                             '    slat (nsec): min=1583, '
                                             'max=180989, avg=1700.89, '
                                             'stdev=355.44\n'
                                             '    clat (nsec): min=1022, '
                                             'max=221516, avg=33718.71, '
                                             'stdev=8534.70\n'
                                             '     lat (usec): min=15, '
                                             'max=223, avg=35.42, stdev= 8.52\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   25],  '
                                             '5.00th=[   31], 10.00th=[   31], '
                                             '20.00th=[   32],\n'
                                             '     | 30.00th=[   32], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   32], '
                                             '80.00th=[   32], 90.00th=[   '
                                             '44], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   66], 99.90th=[  '
                                             '130], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=894160, '
                                             'max=898496, per=100.00%, '
                                             'avg=895716.93, stdev=996.91, '
                                             'samples=14\n'
                                             '   iops        : min=223540, '
                                             'max=224624, avg=223929.36, '
                                             'stdev=249.12, samples=14\n'
                                             '  lat (usec)   : 2=0.01%, '
                                             '10=0.01%, 20=0.05%, 50=91.15%, '
                                             '100=8.53%\n'
                                             '  lat (usec)   : 250=0.27%\n'
                                             '  cpu          : usr=48.79%, '
                                             'sys=51.19%, ctx=10, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.1%, 2=0.1%, '
                                             '4=1.5%, 8=98.5%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1567356,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6123MiB '
                                             '(6420MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.output'),
                                   'state': {'begin': 1666727749.1792693,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7925474643707275,
                                             'end': 1666727749.9718168,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_387.state')},
                       'cmd_388': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-xnvme_0.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.output'),
                                   'state': {'begin': 1666727750.9704022,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7865891456604004,
                                             'end': 1666727751.7569914,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_388.state')},
                       'cmd_389': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.output'),
                                   'state': {'begin': 1666727751.7590091,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-xnvme_0.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.096212387084961,
                                             'end': 1666727762.8552215,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_389.state')},
                       'cmd_39': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733172,\n'
                                            '  "timestamp_ms" : '
                                            '1666733172034,\n'
                                            '  "time" : "Tue Oct 25 23:26:12 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2409836544,\n'
                                            '        "io_kbytes" : 2353356,\n'
                                            '        "bw_bytes" : 344213190,\n'
                                            '        "bw" : 336145,\n'
                                            '        "iops" : 84036.423368,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 588339,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 10810,\n'
                                            '          "mean" : 173.542789,\n'
                                            '          "stddev" : 22.695499,\n'
                                            '          "N" : 588339\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 10410,\n'
                                            '          "max" : 78336,\n'
                                            '          "mean" : 11484.418524,\n'
                                            '          "stddev" : '
                                            '2190.358012,\n'
                                            '          "N" : 588339,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 10944,\n'
                                            '            "5.000000" : 10944,\n'
                                            '            "10.000000" : 11072,\n'
                                            '            "20.000000" : 11072,\n'
                                            '            "30.000000" : 11072,\n'
                                            '            "40.000000" : 11072,\n'
                                            '            "50.000000" : 11072,\n'
                                            '            "60.000000" : 11200,\n'
                                            '            "70.000000" : 11200,\n'
                                            '            "80.000000" : 11328,\n'
                                            '            "90.000000" : 11328,\n'
                                            '            "95.000000" : 11968,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 28288,\n'
                                            '            "99.900000" : 28544,\n'
                                            '            "99.950000" : 43776,\n'
                                            '            "99.990000" : 68096\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 10580,\n'
                                            '          "max" : 78517,\n'
                                            '          "mean" : 11657.961313,\n'
                                            '          "stddev" : '
                                            '2190.707867,\n'
                                            '          "N" : 588339\n'
                                            '        },\n'
                                            '        "bw_min" : 335000,\n'
                                            '        "bw_max" : 337450,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '336330.692308,\n'
                                            '        "bw_dev" : 777.678960,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 83750,\n'
                                            '        "iops_max" : 84362,\n'
                                            '        "iops_mean" : '
                                            '84082.615385,\n'
                                            '        "iops_stddev" : '
                                            '194.345714,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 83.014286,\n'
                                            '      "sys_cpu" : 16.971429,\n'
                                            '      "ctx" : 16,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 98.617124,\n'
                                            '        "50" : 1.344293,\n'
                                            '        "100" : 0.038583,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=84897: Tue '
                                            'Oct 25 23:26:12 2022\n'
                                            '  read: IOPS=84.0k, BW=328MiB/s '
                                            '(344MB/s)(2298MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=10810, avg=173.54, '
                                            'stdev=22.70\n'
                                            '    clat (nsec): min=10410, '
                                            'max=78336, avg=11484.42, '
                                            'stdev=2190.36\n'
                                            '     lat (nsec): min=10580, '
                                            'max=78517, avg=11657.96, '
                                            'stdev=2190.71\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[10944],  '
                                            '5.00th=[10944], 10.00th=[11072], '
                                            '20.00th=[11072],\n'
                                            '     | 30.00th=[11072], '
                                            '40.00th=[11072], 50.00th=[11072], '
                                            '60.00th=[11200],\n'
                                            '     | 70.00th=[11200], '
                                            '80.00th=[11328], 90.00th=[11328], '
                                            '95.00th=[11968],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[28288], 99.90th=[28544], '
                                            '99.95th=[43776],\n'
                                            '     | 99.99th=[68096]\n'
                                            '   bw (  KiB/s): min=335000, '
                                            'max=337450, per=100.00%, '
                                            'avg=336330.69, stdev=777.68, '
                                            'samples=13\n'
                                            '   iops        : min=83750, '
                                            'max=84362, avg=84082.62, '
                                            'stdev=194.35, samples=13\n'
                                            '  lat (usec)   : 20=98.62%, '
                                            '50=1.34%, 100=0.04%\n'
                                            '  cpu          : usr=83.01%, '
                                            'sys=16.97%, ctx=16, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=588339,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=328MiB/s (344MB/s), '
                                            '328MiB/s-328MiB/s '
                                            '(344MB/s-344MB/s), io=2298MiB '
                                            '(2410MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.output'),
                                  'state': {'begin': 1666726082.608158,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7873132228851318,
                                            'end': 1666726083.3954713,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_39.state')},
                       'cmd_390': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734852,\n'
                                             '  "timestamp_ms" : '
                                             '1666734852329,\n'
                                             '  "time" : "Tue Oct 25 23:54:12 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6413660160,\n'
                                             '        "io_kbytes" : 6263340,\n'
                                             '        "bw_bytes" : 916106293,\n'
                                             '        "bw" : 894635,\n'
                                             '        "iops" : 223657.620340,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1565827,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1463,\n'
                                             '          "max" : 42490,\n'
                                             '          "mean" : 1879.747679,\n'
                                             '          "stddev" : '
                                             '476.780147,\n'
                                             '          "N" : 1565828\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 2766,\n'
                                             '          "max" : 250209,\n'
                                             '          "mean" : '
                                             '33625.378656,\n'
                                             '          "stddev" : '
                                             '9412.417229,\n'
                                             '          "N" : 1565834,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 22912,\n'
                                             '            "5.000000" : 25984,\n'
                                             '            "10.000000" : '
                                             '27264,\n'
                                             '            "20.000000" : '
                                             '28544,\n'
                                             '            "30.000000" : '
                                             '29312,\n'
                                             '            "40.000000" : '
                                             '30336,\n'
                                             '            "50.000000" : '
                                             '31104,\n'
                                             '            "60.000000" : '
                                             '31872,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '36608,\n'
                                             '            "90.000000" : '
                                             '45312,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '63232,\n'
                                             '            "99.500000" : '
                                             '69120,\n'
                                             '            "99.900000" : '
                                             '134144,\n'
                                             '            "99.950000" : '
                                             '142336,\n'
                                             '            "99.990000" : '
                                             '160768\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15689,\n'
                                             '          "max" : 262933,\n'
                                             '          "mean" : '
                                             '35505.126049,\n'
                                             '          "stddev" : '
                                             '9396.777144,\n'
                                             '          "N" : 1565834\n'
                                             '        },\n'
                                             '        "bw_min" : 887864,\n'
                                             '        "bw_max" : 897378,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895042.928571,\n'
                                             '        "bw_dev" : 2300.390016,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 221966,\n'
                                             '        "iops_max" : 224344,\n'
                                             '        "iops_mean" : '
                                             '223760.500000,\n'
                                             '        "iops_stddev" : '
                                             '574.996020,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 22.085714,\n'
                                             '      "sys_cpu" : 54.371429,\n'
                                             '      "ctx" : 539751,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.414669,\n'
                                             '        "50" : 92.437734,\n'
                                             '        "100" : 6.875664,\n'
                                             '        "250" : 0.272188,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=90051: Tue '
                                             'Oct 25 23:54:12 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(916MB/s)(6117MiB/7001msec)\n'
                                             '    slat (nsec): min=1463, '
                                             'max=42490, avg=1879.75, '
                                             'stdev=476.78\n'
                                             '    clat (usec): min=2, max=250, '
                                             'avg=33.63, stdev= 9.41\n'
                                             '     lat (usec): min=15, '
                                             'max=262, avg=35.51, stdev= 9.40\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   23],  '
                                             '5.00th=[   26], 10.00th=[   28], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   37], 90.00th=[   '
                                             '46], 95.00th=[   53],\n'
                                             '     | 99.00th=[   64], '
                                             '99.50th=[   70], 99.90th=[  '
                                             '135], 99.95th=[  143],\n'
                                             '     | 99.99th=[  161]\n'
                                             '   bw (  KiB/s): min=887864, '
                                             'max=897378, per=100.00%, '
                                             'avg=895042.93, stdev=2300.39, '
                                             'samples=14\n'
                                             '   iops        : min=221966, '
                                             'max=224344, avg=223760.50, '
                                             'stdev=575.00, samples=14\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '20=0.41%, 50=92.44%, 100=6.88%, '
                                             '250=0.27%\n'
                                             '  lat (usec)   : 500=0.01%\n'
                                             '  cpu          : usr=22.09%, '
                                             'sys=54.37%, ctx=539751, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1565827,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (916MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(916MB/s-916MB/s), io=6117MiB '
                                             '(6414MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.output'),
                                   'state': {'begin': 1666727762.8579476,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_0.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7954163551330566,
                                             'end': 1666727763.653364,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_390.state')},
                       'cmd_391': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-xnvme_1.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.output'),
                                   'state': {'begin': 1666727764.653474,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.784369945526123,
                                             'end': 1666727765.437844,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_391.state')},
                       'cmd_392': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.output'),
                                   'state': {'begin': 1666727765.4397173,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-xnvme_1.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.105721712112427,
                                             'end': 1666727776.545439,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_392.state')},
                       'cmd_393': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734866,\n'
                                             '  "timestamp_ms" : '
                                             '1666734866018,\n'
                                             '  "time" : "Tue Oct 25 23:54:26 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6422921216,\n'
                                             '        "io_kbytes" : 6272384,\n'
                                             '        "bw_bytes" : 917429112,\n'
                                             '        "bw" : 895926,\n'
                                             '        "iops" : 223980.717040,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1568089,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1482,\n'
                                             '          "max" : 25568,\n'
                                             '          "mean" : 1774.253431,\n'
                                             '          "stddev" : '
                                             '161.376328,\n'
                                             '          "N" : 1568089\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 13826,\n'
                                             '          "max" : 201578,\n'
                                             '          "mean" : '
                                             '33675.984157,\n'
                                             '          "stddev" : '
                                             '9110.646423,\n'
                                             '          "N" : 1568096,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23936,\n'
                                             '            "5.000000" : 25984,\n'
                                             '            "10.000000" : '
                                             '27264,\n'
                                             '            "20.000000" : '
                                             '29056,\n'
                                             '            "30.000000" : '
                                             '30080,\n'
                                             '            "40.000000" : '
                                             '30848,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '32128,\n'
                                             '            "70.000000" : '
                                             '33024,\n'
                                             '            "80.000000" : '
                                             '35072,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '52480,\n'
                                             '            "99.000000" : '
                                             '62720,\n'
                                             '            "99.500000" : '
                                             '68096,\n'
                                             '            "99.900000" : '
                                             '130560,\n'
                                             '            "99.950000" : '
                                             '142336,\n'
                                             '            "99.990000" : '
                                             '158720\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 15499,\n'
                                             '          "max" : 207278,\n'
                                             '          "mean" : '
                                             '35450.237329,\n'
                                             '          "stddev" : '
                                             '9103.650064,\n'
                                             '          "N" : 1568096\n'
                                             '        },\n'
                                             '        "bw_min" : 894984,\n'
                                             '        "bw_max" : 897680,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '896217.000000,\n'
                                             '        "bw_dev" : 865.985877,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223746,\n'
                                             '        "iops_max" : 224420,\n'
                                             '        "iops_mean" : '
                                             '224054.214286,\n'
                                             '        "iops_stddev" : '
                                             '216.539276,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 19.214286,\n'
                                             '      "sys_cpu" : 54.914286,\n'
                                             '      "ctx" : 652730,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.291055,\n'
                                             '        "50" : 92.258539,\n'
                                             '        "100" : 7.203864,\n'
                                             '        "250" : 0.246989,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=90091: Tue '
                                             'Oct 25 23:54:26 2022\n'
                                             '  read: IOPS=224k, BW=875MiB/s '
                                             '(917MB/s)(6125MiB/7001msec)\n'
                                             '    slat (nsec): min=1482, '
                                             'max=25568, avg=1774.25, '
                                             'stdev=161.38\n'
                                             '    clat (usec): min=13, '
                                             'max=201, avg=33.68, stdev= 9.11\n'
                                             '     lat (usec): min=15, '
                                             'max=207, avg=35.45, stdev= 9.10\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   26], 10.00th=[   28], '
                                             '20.00th=[   30],\n'
                                             '     | 30.00th=[   31], '
                                             '40.00th=[   31], 50.00th=[   '
                                             '32], 60.00th=[   33],\n'
                                             '     | 70.00th=[   34], '
                                             '80.00th=[   36], 90.00th=[   '
                                             '45], 95.00th=[   53],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   69], 99.90th=[  '
                                             '131], 99.95th=[  143],\n'
                                             '     | 99.99th=[  159]\n'
                                             '   bw (  KiB/s): min=894984, '
                                             'max=897680, per=100.00%, '
                                             'avg=896217.00, stdev=865.99, '
                                             'samples=14\n'
                                             '   iops        : min=223746, '
                                             'max=224420, avg=224054.21, '
                                             'stdev=216.54, samples=14\n'
                                             '  lat (usec)   : 20=0.29%, '
                                             '50=92.26%, 100=7.20%, 250=0.25%\n'
                                             '  cpu          : usr=19.21%, '
                                             'sys=54.91%, ctx=652730, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1568089,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=875MiB/s (917MB/s), '
                                             '875MiB/s-875MiB/s '
                                             '(917MB/s-917MB/s), io=6125MiB '
                                             '(6423MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.output'),
                                   'state': {'begin': 1666727776.5485766,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_1.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7902369499206543,
                                             'end': 1666727777.3388135,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_393.state')},
                       'cmd_394': {'output': 'rm: cannot remove '
                                             "'/tmp/fio-output_4K_8_libaio-xnvme_2.txt': "
                                             'No such file or directory\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.output'),
                                   'state': {'begin': 1666727778.3286088,
                                             'cmd': 'rm '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7883713245391846,
                                             'end': 1666727779.11698,
                                             'err': 1,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_394.state')},
                       'cmd_395': {'output': '',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.output'),
                                   'state': {'begin': 1666727779.1189435,
                                             'cmd': '/opt/fio/bin/fio '
                                                    '--name="libaio-xnvme" '
                                                    '--rw="randread" '
                                                    '--size="12G" '
                                                    '--iodepth="8" --bs="4K" '
                                                    '--direct="1" --thread="1" '
                                                    '--time_based="1" '
                                                    '--runtime="7" '
                                                    '--ramp_time="3" '
                                                    '--norandommap="1" '
                                                    '--allow_file_create="0" '
                                                    '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                    '--xnvme_be="linux" '
                                                    '--xnvme_async="libaio" '
                                                    '--xnvme_sync="nvme" '
                                                    '--xnvme_admin="nvme" '
                                                    '--xnvme_dev_nsid="1" '
                                                    '--filename="/dev/nvme0n1" '
                                                    '--output-format="normal,json" '
                                                    '--output="/tmp/fio-output_4K_8_libaio-xnvme_2.txt"',
                                             'cwd': 'None',
                                             'elapsed': 11.085206508636475,
                                             'end': 1666727790.20415,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_395.state')},
                       'cmd_396': {'output': 'libaio-xnvme: (g=0): '
                                             'rw=randread, bs=(R) 4096B-4096B, '
                                             '(W) 4096B-4096B, (T) '
                                             '4096B-4096B, ioengine=xnvme, '
                                             'iodepth=8\n'
                                             'fio-3.32-dirty\n'
                                             'Starting 1 thread\n'
                                             '{\n'
                                             '  "fio version" : '
                                             '"fio-3.32-dirty",\n'
                                             '  "timestamp" : 1666734879,\n'
                                             '  "timestamp_ms" : '
                                             '1666734879675,\n'
                                             '  "time" : "Tue Oct 25 23:54:39 '
                                             '2022",\n'
                                             '  "jobs" : [\n'
                                             '    {\n'
                                             '      "jobname" : '
                                             '"libaio-xnvme",\n'
                                             '      "groupid" : 0,\n'
                                             '      "error" : 0,\n'
                                             '      "eta" : 0,\n'
                                             '      "elapsed" : 11,\n'
                                             '      "job options" : {\n'
                                             '        "name" : '
                                             '"libaio-xnvme",\n'
                                             '        "rw" : "randread",\n'
                                             '        "size" : "12G",\n'
                                             '        "iodepth" : "8",\n'
                                             '        "bs" : "4K",\n'
                                             '        "direct" : "1",\n'
                                             '        "thread" : "1",\n'
                                             '        "time_based" : "1",\n'
                                             '        "runtime" : "7",\n'
                                             '        "ramp_time" : "3",\n'
                                             '        "norandommap" : "1",\n'
                                             '        "allow_file_create" : '
                                             '"0",\n'
                                             '        "ioengine" : '
                                             '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                             '        "xnvme_be" : "linux",\n'
                                             '        "xnvme_async" : '
                                             '"libaio",\n'
                                             '        "xnvme_sync" : "nvme",\n'
                                             '        "xnvme_admin" : "nvme",\n'
                                             '        "xnvme_dev_nsid" : "1",\n'
                                             '        "filename" : '
                                             '"/dev/nvme0n1"\n'
                                             '      },\n'
                                             '      "read" : {\n'
                                             '        "io_bytes" : '
                                             '6417117184,\n'
                                             '        "io_kbytes" : 6266716,\n'
                                             '        "bw_bytes" : 916600083,\n'
                                             '        "bw" : 895117,\n'
                                             '        "iops" : 223778.317383,\n'
                                             '        "runtime" : 7001,\n'
                                             '        "total_ios" : 1566672,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 1452,\n'
                                             '          "max" : 42480,\n'
                                             '          "mean" : 2043.000890,\n'
                                             '          "stddev" : '
                                             '749.595984,\n'
                                             '          "N" : 1566672\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 440,\n'
                                             '          "max" : 228789,\n'
                                             '          "mean" : '
                                             '33399.843687,\n'
                                             '          "stddev" : '
                                             '8968.879387,\n'
                                             '          "N" : 1566679,\n'
                                             '          "percentile" : {\n'
                                             '            "1.000000" : 23168,\n'
                                             '            "5.000000" : 27008,\n'
                                             '            "10.000000" : '
                                             '27520,\n'
                                             '            "20.000000" : '
                                             '28544,\n'
                                             '            "30.000000" : '
                                             '29824,\n'
                                             '            "40.000000" : '
                                             '31104,\n'
                                             '            "50.000000" : '
                                             '31360,\n'
                                             '            "60.000000" : '
                                             '31616,\n'
                                             '            "70.000000" : '
                                             '32384,\n'
                                             '            "80.000000" : '
                                             '34048,\n'
                                             '            "90.000000" : '
                                             '44800,\n'
                                             '            "95.000000" : '
                                             '51968,\n'
                                             '            "99.000000" : '
                                             '62208,\n'
                                             '            "99.500000" : '
                                             '67072,\n'
                                             '            "99.900000" : '
                                             '132096,\n'
                                             '            "99.950000" : '
                                             '140288,\n'
                                             '            "99.990000" : '
                                             '156672\n'
                                             '          }\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 18515,\n'
                                             '          "max" : 230592,\n'
                                             '          "mean" : '
                                             '35442.843629,\n'
                                             '          "stddev" : '
                                             '8913.795051,\n'
                                             '          "N" : 1566679\n'
                                             '        },\n'
                                             '        "bw_min" : 893096,\n'
                                             '        "bw_max" : 897528,\n'
                                             '        "bw_agg" : 100.000000,\n'
                                             '        "bw_mean" : '
                                             '895350.214286,\n'
                                             '        "bw_dev" : 1168.143970,\n'
                                             '        "bw_samples" : 14,\n'
                                             '        "iops_min" : 223274,\n'
                                             '        "iops_max" : 224382,\n'
                                             '        "iops_mean" : '
                                             '223837.500000,\n'
                                             '        "iops_stddev" : '
                                             '292.004413,\n'
                                             '        "iops_samples" : 14\n'
                                             '      },\n'
                                             '      "write" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "trim" : {\n'
                                             '        "io_bytes" : 0,\n'
                                             '        "io_kbytes" : 0,\n'
                                             '        "bw_bytes" : 0,\n'
                                             '        "bw" : 0,\n'
                                             '        "iops" : 0.000000,\n'
                                             '        "runtime" : 0,\n'
                                             '        "total_ios" : 0,\n'
                                             '        "short_ios" : 0,\n'
                                             '        "drop_ios" : 0,\n'
                                             '        "slat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "clat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        },\n'
                                             '        "bw_min" : 0,\n'
                                             '        "bw_max" : 0,\n'
                                             '        "bw_agg" : 0.000000,\n'
                                             '        "bw_mean" : 0.000000,\n'
                                             '        "bw_dev" : 0.000000,\n'
                                             '        "bw_samples" : 0,\n'
                                             '        "iops_min" : 0,\n'
                                             '        "iops_max" : 0,\n'
                                             '        "iops_mean" : 0.000000,\n'
                                             '        "iops_stddev" : '
                                             '0.000000,\n'
                                             '        "iops_samples" : 0\n'
                                             '      },\n'
                                             '      "sync" : {\n'
                                             '        "total_ios" : 0,\n'
                                             '        "lat_ns" : {\n'
                                             '          "min" : 0,\n'
                                             '          "max" : 0,\n'
                                             '          "mean" : 0.000000,\n'
                                             '          "stddev" : 0.000000,\n'
                                             '          "N" : 0\n'
                                             '        }\n'
                                             '      },\n'
                                             '      "job_runtime" : 7000,\n'
                                             '      "usr_cpu" : 23.857143,\n'
                                             '      "sys_cpu" : 59.200000,\n'
                                             '      "ctx" : 370799,\n'
                                             '      "majf" : 0,\n'
                                             '      "minf" : 0,\n'
                                             '      "iodepth_level" : {\n'
                                             '        "1" : 0.000000,\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "8" : 100.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_submit" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 100.000000,\n'
                                             '        "8" : 0.000000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "iodepth_complete" : {\n'
                                             '        "0" : 0.000000,\n'
                                             '        "4" : 99.999968,\n'
                                             '        "8" : 0.100000,\n'
                                             '        "16" : 0.000000,\n'
                                             '        "32" : 0.000000,\n'
                                             '        "64" : 0.000000,\n'
                                             '        ">=64" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ns" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.010000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_us" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.010000,\n'
                                             '        "10" : 0.010000,\n'
                                             '        "20" : 0.034468,\n'
                                             '        "50" : 92.821854,\n'
                                             '        "100" : 6.875274,\n'
                                             '        "250" : 0.268595,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_ms" : {\n'
                                             '        "2" : 0.000000,\n'
                                             '        "4" : 0.000000,\n'
                                             '        "10" : 0.000000,\n'
                                             '        "20" : 0.000000,\n'
                                             '        "50" : 0.000000,\n'
                                             '        "100" : 0.000000,\n'
                                             '        "250" : 0.000000,\n'
                                             '        "500" : 0.000000,\n'
                                             '        "750" : 0.000000,\n'
                                             '        "1000" : 0.000000,\n'
                                             '        "2000" : 0.000000,\n'
                                             '        ">=2000" : 0.000000\n'
                                             '      },\n'
                                             '      "latency_depth" : 8,\n'
                                             '      "latency_target" : 0,\n'
                                             '      "latency_percentile" : '
                                             '100.000000,\n'
                                             '      "latency_window" : 0\n'
                                             '    }\n'
                                             '  ]\n'
                                             '}\n'
                                             '\n'
                                             'libaio-xnvme: (groupid=0, '
                                             'jobs=1): err= 0: pid=90132: Tue '
                                             'Oct 25 23:54:39 2022\n'
                                             '  read: IOPS=224k, BW=874MiB/s '
                                             '(917MB/s)(6120MiB/7001msec)\n'
                                             '    slat (nsec): min=1452, '
                                             'max=42480, avg=2043.00, '
                                             'stdev=749.60\n'
                                             '    clat (nsec): min=440, '
                                             'max=228789, avg=33399.84, '
                                             'stdev=8968.88\n'
                                             '     lat (usec): min=18, '
                                             'max=230, avg=35.44, stdev= 8.91\n'
                                             '    clat percentiles (usec):\n'
                                             '     |  1.00th=[   24],  '
                                             '5.00th=[   28], 10.00th=[   28], '
                                             '20.00th=[   29],\n'
                                             '     | 30.00th=[   30], '
                                             '40.00th=[   32], 50.00th=[   '
                                             '32], 60.00th=[   32],\n'
                                             '     | 70.00th=[   33], '
                                             '80.00th=[   35], 90.00th=[   '
                                             '45], 95.00th=[   52],\n'
                                             '     | 99.00th=[   63], '
                                             '99.50th=[   68], 99.90th=[  '
                                             '133], 99.95th=[  141],\n'
                                             '     | 99.99th=[  157]\n'
                                             '   bw (  KiB/s): min=893096, '
                                             'max=897528, per=100.00%, '
                                             'avg=895350.21, stdev=1168.14, '
                                             'samples=14\n'
                                             '   iops        : min=223274, '
                                             'max=224382, avg=223837.50, '
                                             'stdev=292.00, samples=14\n'
                                             '  lat (nsec)   : 500=0.01%\n'
                                             '  lat (usec)   : 4=0.01%, '
                                             '10=0.01%, 20=0.03%, 50=92.82%, '
                                             '100=6.88%\n'
                                             '  lat (usec)   : 250=0.27%\n'
                                             '  cpu          : usr=23.86%, '
                                             'sys=59.20%, ctx=370799, majf=0, '
                                             'minf=0\n'
                                             '  IO depths    : 1=0.0%, 2=0.0%, '
                                             '4=0.0%, 8=100.0%, 16=0.0%, '
                                             '32=0.0%, >=64=0.0%\n'
                                             '     submit    : 0=0.0%, '
                                             '4=100.0%, 8=0.0%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     complete  : 0=0.0%, '
                                             '4=100.0%, 8=0.1%, 16=0.0%, '
                                             '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                             '     issued rwts: '
                                             'total=1566672,0,0,0 '
                                             'short=0,0,0,0 dropped=0,0,0,0\n'
                                             '     latency   : target=0, '
                                             'window=0, percentile=100.00%, '
                                             'depth=8\n'
                                             '\n'
                                             'Run status group 0 (all jobs):\n'
                                             '   READ: bw=874MiB/s (917MB/s), '
                                             '874MiB/s-874MiB/s '
                                             '(917MB/s-917MB/s), io=6120MiB '
                                             '(6417MB), run=7001-7001msec\n',
                                   'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.output'),
                                   'state': {'begin': 1666727790.2068746,
                                             'cmd': 'cat '
                                                    '/tmp/fio-output_4K_8_libaio-xnvme_2.txt',
                                             'cwd': 'None',
                                             'elapsed': 0.7910661697387695,
                                             'end': 1666727790.9979408,
                                             'err': 0,
                                             'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                             'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.output'},
                                   'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_396.state')},
                       'cmd_40': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.output'),
                                  'state': {'begin': 1666726085.471208,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7902677059173584,
                                            'end': 1666726086.2614758,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_40.state')},
                       'cmd_41': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.output'),
                                  'state': {'begin': 1666726086.2634542,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.455034732818604,
                                            'end': 1666726097.718489,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_41.state')},
                       'cmd_42': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733187,\n'
                                            '  "timestamp_ms" : '
                                            '1666733187146,\n'
                                            '  "time" : "Tue Oct 25 23:26:27 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2400022528,\n'
                                            '        "io_kbytes" : 2343772,\n'
                                            '        "bw_bytes" : 342811388,\n'
                                            '        "bw" : 334776,\n'
                                            '        "iops" : 83694.186545,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 585943,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 160,\n'
                                            '          "max" : 9938,\n'
                                            '          "mean" : 176.388493,\n'
                                            '          "stddev" : 26.408079,\n'
                                            '          "N" : 585943\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 10430,\n'
                                            '          "max" : 113473,\n'
                                            '          "mean" : 11527.381930,\n'
                                            '          "stddev" : '
                                            '2234.725223,\n'
                                            '          "N" : 585943,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 10944,\n'
                                            '            "5.000000" : 11072,\n'
                                            '            "10.000000" : 11072,\n'
                                            '            "20.000000" : 11072,\n'
                                            '            "30.000000" : 11072,\n'
                                            '            "40.000000" : 11072,\n'
                                            '            "50.000000" : 11200,\n'
                                            '            "60.000000" : 11200,\n'
                                            '            "70.000000" : 11328,\n'
                                            '            "80.000000" : 11328,\n'
                                            '            "90.000000" : 11456,\n'
                                            '            "95.000000" : 12096,\n'
                                            '            "99.000000" : 22400,\n'
                                            '            "99.500000" : 28288,\n'
                                            '            "99.900000" : 28800,\n'
                                            '            "99.950000" : 44288,\n'
                                            '            "99.990000" : 68096\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 10610,\n'
                                            '          "max" : 113653,\n'
                                            '          "mean" : 11703.770423,\n'
                                            '          "stddev" : '
                                            '2235.118972,\n'
                                            '          "N" : 585943\n'
                                            '        },\n'
                                            '        "bw_min" : 334472,\n'
                                            '        "bw_max" : 336208,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '334961.142857,\n'
                                            '        "bw_dev" : 434.449228,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 83618,\n'
                                            '        "iops_max" : 84052,\n'
                                            '        "iops_mean" : '
                                            '83740.214286,\n'
                                            '        "iops_stddev" : '
                                            '108.614476,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 80.728571,\n'
                                            '      "sys_cpu" : 19.257143,\n'
                                            '      "ctx" : 15,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 98.614541,\n'
                                            '        "50" : 1.340915,\n'
                                            '        "100" : 0.044373,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=84947: Tue '
                                            'Oct 25 23:26:27 2022\n'
                                            '  read: IOPS=83.7k, BW=327MiB/s '
                                            '(343MB/s)(2289MiB/7001msec)\n'
                                            '    slat (nsec): min=160, '
                                            'max=9938, avg=176.39, '
                                            'stdev=26.41\n'
                                            '    clat (usec): min=10, max=113, '
                                            'avg=11.53, stdev= 2.23\n'
                                            '     lat (usec): min=10, max=113, '
                                            'avg=11.70, stdev= 2.24\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[10944],  '
                                            '5.00th=[11072], 10.00th=[11072], '
                                            '20.00th=[11072],\n'
                                            '     | 30.00th=[11072], '
                                            '40.00th=[11072], 50.00th=[11200], '
                                            '60.00th=[11200],\n'
                                            '     | 70.00th=[11328], '
                                            '80.00th=[11328], 90.00th=[11456], '
                                            '95.00th=[12096],\n'
                                            '     | 99.00th=[22400], '
                                            '99.50th=[28288], 99.90th=[28800], '
                                            '99.95th=[44288],\n'
                                            '     | 99.99th=[68096]\n'
                                            '   bw (  KiB/s): min=334472, '
                                            'max=336208, per=100.00%, '
                                            'avg=334961.14, stdev=434.45, '
                                            'samples=14\n'
                                            '   iops        : min=83618, '
                                            'max=84052, avg=83740.21, '
                                            'stdev=108.61, samples=14\n'
                                            '  lat (usec)   : 20=98.61%, '
                                            '50=1.34%, 100=0.04%, 250=0.01%\n'
                                            '  cpu          : usr=80.73%, '
                                            'sys=19.26%, ctx=15, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=585943,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=327MiB/s (343MB/s), '
                                            '327MiB/s-327MiB/s '
                                            '(343MB/s-343MB/s), io=2289MiB '
                                            '(2400MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.output'),
                                  'state': {'begin': 1666726097.7213132,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7839422225952148,
                                            'end': 1666726098.5052555,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_42.state')},
                       'cmd_43': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.output'),
                                  'state': {'begin': 1666726100.5752542,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.790607213973999,
                                            'end': 1666726101.3658614,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_43.state')},
                       'cmd_44': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.output'),
                                  'state': {'begin': 1666726101.367674,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_uring" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_uring_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.460201978683472,
                                            'end': 1666726112.827876,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_44.state')},
                       'cmd_45': {'output': 'io_uring-bdev_uring: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733202,\n'
                                            '  "timestamp_ms" : '
                                            '1666733202256,\n'
                                            '  "time" : "Tue Oct 25 23:26:42 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_uring",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_uring",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_uring_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2856644608,\n'
                                            '        "io_kbytes" : 2789692,\n'
                                            '        "bw_bytes" : 408033796,\n'
                                            '        "bw" : 398470,\n'
                                            '        "iops" : 99617.626053,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 697423,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 161,\n'
                                            '          "max" : 12935,\n'
                                            '          "mean" : 181.008962,\n'
                                            '          "stddev" : 43.708701,\n'
                                            '          "N" : 697423\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 9037,\n'
                                            '          "max" : 95539,\n'
                                            '          "mean" : 9612.561520,\n'
                                            '          "stddev" : '
                                            '1943.039396,\n'
                                            '          "N" : 697423,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 9152,\n'
                                            '            "5.000000" : 9152,\n'
                                            '            "10.000000" : 9152,\n'
                                            '            "20.000000" : 9152,\n'
                                            '            "30.000000" : 9280,\n'
                                            '            "40.000000" : 9280,\n'
                                            '            "50.000000" : 9280,\n'
                                            '            "60.000000" : 9408,\n'
                                            '            "70.000000" : 9408,\n'
                                            '            "80.000000" : 9408,\n'
                                            '            "90.000000" : 9408,\n'
                                            '            "95.000000" : 10048,\n'
                                            '            "99.000000" : 18816,\n'
                                            '            "99.500000" : 20864,\n'
                                            '            "99.900000" : 25472,\n'
                                            '            "99.950000" : 42752,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9218,\n'
                                            '          "max" : 95719,\n'
                                            '          "mean" : 9793.570482,\n'
                                            '          "stddev" : '
                                            '1943.679350,\n'
                                            '          "N" : 697423\n'
                                            '        },\n'
                                            '        "bw_min" : 397920,\n'
                                            '        "bw_max" : 399024,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '398570.642857,\n'
                                            '        "bw_dev" : 358.930807,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 99480,\n'
                                            '        "iops_max" : 99756,\n'
                                            '        "iops_mean" : '
                                            '99642.642857,\n'
                                            '        "iops_stddev" : '
                                            '89.712024,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 70.271429,\n'
                                            '      "sys_cpu" : 29.700000,\n'
                                            '      "ctx" : 8,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 94.961594,\n'
                                            '        "20" : 4.229141,\n'
                                            '        "50" : 0.766393,\n'
                                            '        "100" : 0.042872,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_uring: (groupid=0, '
                                            'jobs=1): err= 0: pid=84996: Tue '
                                            'Oct 25 23:26:42 2022\n'
                                            '  read: IOPS=99.6k, BW=389MiB/s '
                                            '(408MB/s)(2724MiB/7001msec)\n'
                                            '    slat (nsec): min=161, '
                                            'max=12935, avg=181.01, '
                                            'stdev=43.71\n'
                                            '    clat (nsec): min=9037, '
                                            'max=95539, avg=9612.56, '
                                            'stdev=1943.04\n'
                                            '     lat (nsec): min=9218, '
                                            'max=95719, avg=9793.57, '
                                            'stdev=1943.68\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 9152],  5.00th=[ '
                                            '9152], 10.00th=[ 9152], 20.00th=[ '
                                            '9152],\n'
                                            '     | 30.00th=[ 9280], 40.00th=[ '
                                            '9280], 50.00th=[ 9280], 60.00th=[ '
                                            '9408],\n'
                                            '     | 70.00th=[ 9408], 80.00th=[ '
                                            '9408], 90.00th=[ 9408], '
                                            '95.00th=[10048],\n'
                                            '     | 99.00th=[18816], '
                                            '99.50th=[20864], 99.90th=[25472], '
                                            '99.95th=[42752],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=397920, '
                                            'max=399024, per=100.00%, '
                                            'avg=398570.64, stdev=358.93, '
                                            'samples=14\n'
                                            '   iops        : min=99480, '
                                            'max=99756, avg=99642.64, '
                                            'stdev=89.71, samples=14\n'
                                            '  lat (usec)   : 10=94.96%, '
                                            '20=4.23%, 50=0.77%, 100=0.04%\n'
                                            '  cpu          : usr=70.27%, '
                                            'sys=29.70%, ctx=8, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=697423,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=389MiB/s (408MB/s), '
                                            '389MiB/s-389MiB/s '
                                            '(408MB/s-408MB/s), io=2724MiB '
                                            '(2857MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.output'),
                                  'state': {'begin': 1666726112.8305573,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7889697551727295,
                                            'end': 1666726113.619527,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_45.state')},
                       'cmd_46': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.output'),
                                  'state': {'begin': 1666726115.6897912,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7863481044769287,
                                            'end': 1666726116.4761393,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_46.state')},
                       'cmd_47': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.output'),
                                  'state': {'begin': 1666726116.478392,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.34797716140747,
                                            'end': 1666726127.826369,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_47.state')},
                       'cmd_48': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733217,\n'
                                            '  "timestamp_ms" : '
                                            '1666733217255,\n'
                                            '  "time" : "Tue Oct 25 23:26:57 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2650079232,\n'
                                            '        "io_kbytes" : 2587968,\n'
                                            '        "bw_bytes" : 378528671,\n'
                                            '        "bw" : 369656,\n'
                                            '        "iops" : 92414.226539,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 646992,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1673,\n'
                                            '          "max" : 15529,\n'
                                            '          "mean" : 1800.539421,\n'
                                            '          "stddev" : 116.222659,\n'
                                            '          "N" : 646992\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 251,\n'
                                            '          "max" : 115536,\n'
                                            '          "mean" : 8778.187362,\n'
                                            '          "stddev" : '
                                            '1981.005506,\n'
                                            '          "N" : 646992,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8256,\n'
                                            '            "5.000000" : 8384,\n'
                                            '            "10.000000" : 8384,\n'
                                            '            "20.000000" : 8384,\n'
                                            '            "30.000000" : 8384,\n'
                                            '            "40.000000" : 8384,\n'
                                            '            "50.000000" : 8512,\n'
                                            '            "60.000000" : 8512,\n'
                                            '            "70.000000" : 8512,\n'
                                            '            "80.000000" : 8512,\n'
                                            '            "90.000000" : 8640,\n'
                                            '            "95.000000" : 9408,\n'
                                            '            "99.000000" : 18816,\n'
                                            '            "99.500000" : 20352,\n'
                                            '            "99.900000" : 24960,\n'
                                            '            "99.950000" : 41728,\n'
                                            '            "99.990000" : 66048\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9598,\n'
                                            '          "max" : 117290,\n'
                                            '          "mean" : 10578.726783,\n'
                                            '          "stddev" : '
                                            '1983.792726,\n'
                                            '          "N" : 646992\n'
                                            '        },\n'
                                            '        "bw_min" : 367720,\n'
                                            '        "bw_max" : 370808,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '369810.769231,\n'
                                            '        "bw_dev" : 877.410504,\n'
                                            '        "bw_samples" : 13,\n'
                                            '        "iops_min" : 91930,\n'
                                            '        "iops_max" : 92702,\n'
                                            '        "iops_mean" : '
                                            '92452.615385,\n'
                                            '        "iops_stddev" : '
                                            '219.323406,\n'
                                            '        "iops_samples" : 13\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 5.642857,\n'
                                            '      "sys_cpu" : 94.328571,\n'
                                            '      "ctx" : 14,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 95.804121,\n'
                                            '        "20" : 3.551049,\n'
                                            '        "50" : 0.603562,\n'
                                            '        "100" : 0.036940,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85046: Tue '
                                            'Oct 25 23:26:57 2022\n'
                                            '  read: IOPS=92.4k, BW=361MiB/s '
                                            '(379MB/s)(2527MiB/7001msec)\n'
                                            '    slat (nsec): min=1673, '
                                            'max=15529, avg=1800.54, '
                                            'stdev=116.22\n'
                                            '    clat (nsec): min=251, '
                                            'max=115536, avg=8778.19, '
                                            'stdev=1981.01\n'
                                            '     lat (usec): min=9, max=117, '
                                            'avg=10.58, stdev= 1.98\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8256],  5.00th=[ '
                                            '8384], 10.00th=[ 8384], 20.00th=[ '
                                            '8384],\n'
                                            '     | 30.00th=[ 8384], 40.00th=[ '
                                            '8384], 50.00th=[ 8512], 60.00th=[ '
                                            '8512],\n'
                                            '     | 70.00th=[ 8512], 80.00th=[ '
                                            '8512], 90.00th=[ 8640], 95.00th=[ '
                                            '9408],\n'
                                            '     | 99.00th=[18816], '
                                            '99.50th=[20352], 99.90th=[24960], '
                                            '99.95th=[41728],\n'
                                            '     | 99.99th=[66048]\n'
                                            '   bw (  KiB/s): min=367720, '
                                            'max=370808, per=100.00%, '
                                            'avg=369810.77, stdev=877.41, '
                                            'samples=13\n'
                                            '   iops        : min=91930, '
                                            'max=92702, avg=92452.62, '
                                            'stdev=219.32, samples=13\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=95.80%, 20=3.55%, 50=0.60%, '
                                            '100=0.04%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=5.64%, '
                                            'sys=94.33%, ctx=14, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=646992,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=361MiB/s (379MB/s), '
                                            '361MiB/s-361MiB/s '
                                            '(379MB/s-379MB/s), io=2527MiB '
                                            '(2650MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.output'),
                                  'state': {'begin': 1666726127.829114,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7902984619140625,
                                            'end': 1666726128.6194124,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_48.state')},
                       'cmd_49': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.output'),
                                  'state': {'begin': 1666726130.6240525,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7834982872009277,
                                            'end': 1666726131.4075508,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_49.state')},
                       'cmd_50': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.output'),
                                  'state': {'begin': 1666726131.4095988,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.450175523757935,
                                            'end': 1666726142.8597744,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_50.state')},
                       'cmd_51': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733232,\n'
                                            '  "timestamp_ms" : '
                                            '1666733232288,\n'
                                            '  "time" : "Tue Oct 25 23:27:12 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2934267904,\n'
                                            '        "io_kbytes" : 2865496,\n'
                                            '        "bw_bytes" : 419121254,\n'
                                            '        "bw" : 409298,\n'
                                            '        "iops" : 102324.525068,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 716374,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1543,\n'
                                            '          "max" : 18425,\n'
                                            '          "mean" : 1610.136684,\n'
                                            '          "stddev" : 115.405056,\n'
                                            '          "N" : 716374\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 400,\n'
                                            '          "max" : 1295729,\n'
                                            '          "mean" : 7923.244064,\n'
                                            '          "stddev" : '
                                            '2451.556179,\n'
                                            '          "N" : 716374,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7392,\n'
                                            '            "5.000000" : 7456,\n'
                                            '            "10.000000" : 7456,\n'
                                            '            "20.000000" : 7520,\n'
                                            '            "30.000000" : 7520,\n'
                                            '            "40.000000" : 7584,\n'
                                            '            "50.000000" : 7648,\n'
                                            '            "60.000000" : 7648,\n'
                                            '            "70.000000" : 7712,\n'
                                            '            "80.000000" : 7712,\n'
                                            '            "90.000000" : 7776,\n'
                                            '            "95.000000" : 8160,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 20096,\n'
                                            '            "99.900000" : 22656,\n'
                                            '            "99.950000" : 32384,\n'
                                            '            "99.990000" : 65280\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8926,\n'
                                            '          "max" : 1297322,\n'
                                            '          "mean" : 9533.380748,\n'
                                            '          "stddev" : '
                                            '2454.613923,\n'
                                            '          "N" : 716374\n'
                                            '        },\n'
                                            '        "bw_min" : 408576,\n'
                                            '        "bw_max" : 410869,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '409391.785714,\n'
                                            '        "bw_dev" : 583.681180,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 102144,\n'
                                            '        "iops_max" : 102717,\n'
                                            '        "iops_mean" : '
                                            '102347.928571,\n'
                                            '        "iops_stddev" : '
                                            '145.871632,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 6.957143,\n'
                                            '      "sys_cpu" : 93.000000,\n'
                                            '      "ctx" : 48,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 96.128419,\n'
                                            '        "20" : 3.324520,\n'
                                            '        "50" : 0.511325,\n'
                                            '        "100" : 0.034619,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85097: Tue '
                                            'Oct 25 23:27:12 2022\n'
                                            '  read: IOPS=102k, BW=400MiB/s '
                                            '(419MB/s)(2798MiB/7001msec)\n'
                                            '    slat (nsec): min=1543, '
                                            'max=18425, avg=1610.14, '
                                            'stdev=115.41\n'
                                            '    clat (nsec): min=400, '
                                            'max=1295.7k, avg=7923.24, '
                                            'stdev=2451.56\n'
                                            '     lat (usec): min=8, max=1297, '
                                            'avg= 9.53, stdev= 2.45\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7392],  5.00th=[ '
                                            '7456], 10.00th=[ 7456], 20.00th=[ '
                                            '7520],\n'
                                            '     | 30.00th=[ 7520], 40.00th=[ '
                                            '7584], 50.00th=[ 7648], 60.00th=[ '
                                            '7648],\n'
                                            '     | 70.00th=[ 7712], 80.00th=[ '
                                            '7712], 90.00th=[ 7776], 95.00th=[ '
                                            '8160],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[20096], 99.90th=[22656], '
                                            '99.95th=[32384],\n'
                                            '     | 99.99th=[65280]\n'
                                            '   bw (  KiB/s): min=408576, '
                                            'max=410869, per=100.00%, '
                                            'avg=409391.79, stdev=583.68, '
                                            'samples=14\n'
                                            '   iops        : min=102144, '
                                            'max=102717, avg=102347.93, '
                                            'stdev=145.87, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=96.13%, 20=3.32%, 50=0.51%, '
                                            '100=0.03%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  lat (msec)   : 2=0.01%\n'
                                            '  cpu          : usr=6.96%, '
                                            'sys=93.00%, ctx=48, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=716374,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=400MiB/s (419MB/s), '
                                            '400MiB/s-400MiB/s '
                                            '(419MB/s-419MB/s), io=2798MiB '
                                            '(2934MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.output'),
                                  'state': {'begin': 1666726142.863593,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7923583984375,
                                            'end': 1666726143.6559515,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_51.state')},
                       'cmd_52': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.output'),
                                  'state': {'begin': 1666726145.6664793,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7846190929412842,
                                            'end': 1666726146.4510984,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_52.state')},
                       'cmd_53': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.output'),
                                  'state': {'begin': 1666726146.4534276,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.453392505645752,
                                            'end': 1666726157.90682,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_53.state')},
                       'cmd_54': {'output': 'io_uring-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733247,\n'
                                            '  "timestamp_ms" : '
                                            '1666733247334,\n'
                                            '  "time" : "Tue Oct 25 23:27:27 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_io_uring.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2939764736,\n'
                                            '        "io_kbytes" : 2870864,\n'
                                            '        "bw_bytes" : 419906404,\n'
                                            '        "bw" : 410064,\n'
                                            '        "iops" : 102516.211970,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 717716,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1522,\n'
                                            '          "max" : 14347,\n'
                                            '          "mean" : 1578.864615,\n'
                                            '          "stddev" : 120.389006,\n'
                                            '          "N" : 717716\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 241,\n'
                                            '          "max" : 96561,\n'
                                            '          "mean" : 7938.478975,\n'
                                            '          "stddev" : '
                                            '1946.448668,\n'
                                            '          "N" : 717716,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7456,\n'
                                            '            "5.000000" : 7456,\n'
                                            '            "10.000000" : 7456,\n'
                                            '            "20.000000" : 7520,\n'
                                            '            "30.000000" : 7520,\n'
                                            '            "40.000000" : 7584,\n'
                                            '            "50.000000" : 7648,\n'
                                            '            "60.000000" : 7648,\n'
                                            '            "70.000000" : 7712,\n'
                                            '            "80.000000" : 7712,\n'
                                            '            "90.000000" : 7776,\n'
                                            '            "95.000000" : 8160,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 20352,\n'
                                            '            "99.900000" : 22912,\n'
                                            '            "99.950000" : 32640,\n'
                                            '            "99.990000" : 65280\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 8926,\n'
                                            '          "max" : 98124,\n'
                                            '          "mean" : 9517.343590,\n'
                                            '          "stddev" : '
                                            '1950.790720,\n'
                                            '          "N" : 717716\n'
                                            '        },\n'
                                            '        "bw_min" : 409424,\n'
                                            '        "bw_max" : 411160,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '410161.928571,\n'
                                            '        "bw_dev" : 477.509318,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 102354,\n'
                                            '        "iops_max" : 102790,\n'
                                            '        "iops_mean" : '
                                            '102540.428571,\n'
                                            '        "iops_stddev" : '
                                            '119.741204,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 8.157143,\n'
                                            '      "sys_cpu" : 91.814286,\n'
                                            '      "ctx" : 5,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 96.122422,\n'
                                            '        "20" : 3.278456,\n'
                                            '        "50" : 0.559135,\n'
                                            '        "100" : 0.038316,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85148: Tue '
                                            'Oct 25 23:27:27 2022\n'
                                            '  read: IOPS=103k, BW=400MiB/s '
                                            '(420MB/s)(2804MiB/7001msec)\n'
                                            '    slat (nsec): min=1522, '
                                            'max=14347, avg=1578.86, '
                                            'stdev=120.39\n'
                                            '    clat (nsec): min=241, '
                                            'max=96561, avg=7938.48, '
                                            'stdev=1946.45\n'
                                            '     lat (nsec): min=8926, '
                                            'max=98124, avg=9517.34, '
                                            'stdev=1950.79\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7456],  5.00th=[ '
                                            '7456], 10.00th=[ 7456], 20.00th=[ '
                                            '7520],\n'
                                            '     | 30.00th=[ 7520], 40.00th=[ '
                                            '7584], 50.00th=[ 7648], 60.00th=[ '
                                            '7648],\n'
                                            '     | 70.00th=[ 7712], 80.00th=[ '
                                            '7712], 90.00th=[ 7776], 95.00th=[ '
                                            '8160],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[20352], 99.90th=[22912], '
                                            '99.95th=[32640],\n'
                                            '     | 99.99th=[65280]\n'
                                            '   bw (  KiB/s): min=409424, '
                                            'max=411160, per=100.00%, '
                                            'avg=410161.93, stdev=477.51, '
                                            'samples=14\n'
                                            '   iops        : min=102354, '
                                            'max=102790, avg=102540.43, '
                                            'stdev=119.74, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%, 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=96.12%, 20=3.28%, 50=0.56%, '
                                            '100=0.04%\n'
                                            '  cpu          : usr=8.16%, '
                                            'sys=91.81%, ctx=5, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=717716,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=400MiB/s (420MB/s), '
                                            '400MiB/s-400MiB/s '
                                            '(420MB/s-420MB/s), io=2804MiB '
                                            '(2940MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.output'),
                                  'state': {'begin': 1666726157.9090652,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7853574752807617,
                                            'end': 1666726158.6944227,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_54.state')},
                       'cmd_55': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.output'),
                                  'state': {'begin': 1666726159.6800706,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.782827615737915,
                                            'end': 1666726160.4628983,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_55.state')},
                       'cmd_56': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.output'),
                                  'state': {'begin': 1666726160.4650114,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.04825758934021,
                                            'end': 1666726171.513269,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_56.state')},
                       'cmd_57': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733261,\n'
                                            '  "timestamp_ms" : '
                                            '1666733261012,\n'
                                            '  "time" : "Tue Oct 25 23:27:41 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1795207168,\n'
                                            '        "io_kbytes" : 1753132,\n'
                                            '        "bw_bytes" : 256421535,\n'
                                            '        "bw" : 250411,\n'
                                            '        "iops" : 62602.913869,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 438283,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2655,\n'
                                            '          "max" : 27832,\n'
                                            '          "mean" : 2842.061771,\n'
                                            '          "stddev" : 230.543540,\n'
                                            '          "N" : 438283\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 281,\n'
                                            '          "max" : 122980,\n'
                                            '          "mean" : 12554.664060,\n'
                                            '          "stddev" : '
                                            '2139.295770,\n'
                                            '          "N" : 438283,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 12096,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12224,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12224,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 28288,\n'
                                            '            "99.900000" : 40704,\n'
                                            '            "99.950000" : 54016,\n'
                                            '            "99.990000" : 68096\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12182,\n'
                                            '          "max" : 125816,\n'
                                            '          "mean" : 15396.725830,\n'
                                            '          "stddev" : '
                                            '2157.279613,\n'
                                            '          "N" : 438283\n'
                                            '        },\n'
                                            '        "bw_min" : 249896,\n'
                                            '        "bw_max" : 251094,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '250470.142857,\n'
                                            '        "bw_dev" : 335.320250,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 62472,\n'
                                            '        "iops_max" : 62773,\n'
                                            '        "iops_mean" : '
                                            '62617.500000,\n'
                                            '        "iops_stddev" : '
                                            '83.901728,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.585714,\n'
                                            '      "sys_cpu" : 38.442857,\n'
                                            '      "ctx" : 438270,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.596113,\n'
                                            '        "50" : 1.331788,\n'
                                            '        "100" : 0.064798,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85187: Tue '
                                            'Oct 25 23:27:41 2022\n'
                                            '  read: IOPS=62.6k, BW=245MiB/s '
                                            '(256MB/s)(1712MiB/7001msec)\n'
                                            '    slat (nsec): min=2655, '
                                            'max=27832, avg=2842.06, '
                                            'stdev=230.54\n'
                                            '    clat (nsec): min=281, '
                                            'max=122980, avg=12554.66, '
                                            'stdev=2139.30\n'
                                            '     lat (usec): min=12, max=125, '
                                            'avg=15.40, stdev= 2.16\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[12096], 10.00th=[12096], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12224], '
                                            '40.00th=[12224], 50.00th=[12224], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12480], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[28288], 99.90th=[40704], '
                                            '99.95th=[54016],\n'
                                            '     | 99.99th=[68096]\n'
                                            '   bw (  KiB/s): min=249896, '
                                            'max=251094, per=100.00%, '
                                            'avg=250470.14, stdev=335.32, '
                                            'samples=14\n'
                                            '   iops        : min=62472, '
                                            'max=62773, avg=62617.50, '
                                            'stdev=83.90, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.60%, 50=1.33%, '
                                            '100=0.06%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=10.59%, '
                                            'sys=38.44%, ctx=438270, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=438283,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=245MiB/s (256MB/s), '
                                            '245MiB/s-245MiB/s '
                                            '(256MB/s-256MB/s), io=1712MiB '
                                            '(1795MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.output'),
                                  'state': {'begin': 1666726171.5161881,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7912197113037109,
                                            'end': 1666726172.3074079,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_57.state')},
                       'cmd_58': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.output'),
                                  'state': {'begin': 1666726173.3039281,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7914633750915527,
                                            'end': 1666726174.0953915,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_58.state')},
                       'cmd_59': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.output'),
                                  'state': {'begin': 1666726174.097773,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.052335023880005,
                                            'end': 1666726185.150108,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_59.state')},
                       'cmd_60': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733274,\n'
                                            '  "timestamp_ms" : '
                                            '1666733274648,\n'
                                            '  "time" : "Tue Oct 25 23:27:54 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1804898304,\n'
                                            '        "io_kbytes" : 1762596,\n'
                                            '        "bw_bytes" : 257805785,\n'
                                            '        "bw" : 251763,\n'
                                            '        "iops" : 62940.865591,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 440649,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2645,\n'
                                            '          "max" : 27261,\n'
                                            '          "mean" : 2816.685162,\n'
                                            '          "stddev" : 258.194088,\n'
                                            '          "N" : 440649\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 380,\n'
                                            '          "max" : 467586,\n'
                                            '          "mean" : 12522.016850,\n'
                                            '          "stddev" : '
                                            '2175.039916,\n'
                                            '          "N" : 440649,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 12096,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12224,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12224,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 28032,\n'
                                            '            "99.900000" : 33536,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 66048\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12243,\n'
                                            '          "max" : 470382,\n'
                                            '          "mean" : 15338.702012,\n'
                                            '          "stddev" : '
                                            '2192.213103,\n'
                                            '          "N" : 440649\n'
                                            '        },\n'
                                            '        "bw_min" : 250829,\n'
                                            '        "bw_max" : 252600,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '251826.071429,\n'
                                            '        "bw_dev" : 417.064921,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 62707,\n'
                                            '        "iops_max" : 63150,\n'
                                            '        "iops_mean" : '
                                            '62956.500000,\n'
                                            '        "iops_stddev" : '
                                            '104.312216,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.042857,\n'
                                            '      "sys_cpu" : 38.328571,\n'
                                            '      "ctx" : 440630,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.638372,\n'
                                            '        "50" : 1.295589,\n'
                                            '        "100" : 0.056054,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85227: Tue '
                                            'Oct 25 23:27:54 2022\n'
                                            '  read: IOPS=62.9k, BW=246MiB/s '
                                            '(258MB/s)(1721MiB/7001msec)\n'
                                            '    slat (nsec): min=2645, '
                                            'max=27261, avg=2816.69, '
                                            'stdev=258.19\n'
                                            '    clat (nsec): min=380, '
                                            'max=467586, avg=12522.02, '
                                            'stdev=2175.04\n'
                                            '     lat (usec): min=12, max=470, '
                                            'avg=15.34, stdev= 2.19\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[12096], 10.00th=[12096], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12224], '
                                            '40.00th=[12224], 50.00th=[12224], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12480], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[28032], 99.90th=[33536], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[66048]\n'
                                            '   bw (  KiB/s): min=250829, '
                                            'max=252600, per=100.00%, '
                                            'avg=251826.07, stdev=417.06, '
                                            'samples=14\n'
                                            '   iops        : min=62707, '
                                            'max=63150, avg=62956.50, '
                                            'stdev=104.31, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=0.01%, 20=98.64%, 50=1.30%, '
                                            '100=0.06%\n'
                                            '  lat (usec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  cpu          : usr=11.04%, '
                                            'sys=38.33%, ctx=440630, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=440649,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=246MiB/s (258MB/s), '
                                            '246MiB/s-246MiB/s '
                                            '(258MB/s-258MB/s), io=1721MiB '
                                            '(1805MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.output'),
                                  'state': {'begin': 1666726185.1532598,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7865962982177734,
                                            'end': 1666726185.939856,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_60.state')},
                       'cmd_61': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_io_uring-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.output'),
                                  'state': {'begin': 1666726186.9379005,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7873010635375977,
                                            'end': 1666726187.7252016,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_61.state')},
                       'cmd_62': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.output'),
                                  'state': {'begin': 1666726187.7272713,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="io_uring-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="io_uring" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_io_uring-xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.057294368743896,
                                            'end': 1666726198.7845657,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_62.state')},
                       'cmd_63': {'output': 'io_uring-xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733288,\n'
                                            '  "timestamp_ms" : '
                                            '1666733288284,\n'
                                            '  "time" : "Tue Oct 25 23:28:08 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"io_uring-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"io_uring-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"io_uring",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1813983232,\n'
                                            '        "io_kbytes" : 1771468,\n'
                                            '        "bw_bytes" : 259103446,\n'
                                            '        "bw" : 253030,\n'
                                            '        "iops" : 63257.534638,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 442866,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2584,\n'
                                            '          "max" : 28093,\n'
                                            '          "mean" : 2753.253893,\n'
                                            '          "stddev" : 286.498944,\n'
                                            '          "N" : 442867\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 421,\n'
                                            '          "max" : 181340,\n'
                                            '          "mean" : 12481.790944,\n'
                                            '          "stddev" : '
                                            '2028.063473,\n'
                                            '          "N" : 442866,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 11968,\n'
                                            '            "10.000000" : 11968,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12096,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12224,\n'
                                            '            "60.000000" : 12224,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12352,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 27776,\n'
                                            '            "99.900000" : 32384,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 66048\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12102,\n'
                                            '          "max" : 184025,\n'
                                            '          "mean" : 15235.045014,\n'
                                            '          "stddev" : '
                                            '2052.149286,\n'
                                            '          "N" : 442866\n'
                                            '        },\n'
                                            '        "bw_min" : 252120,\n'
                                            '        "bw_max" : 254468,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '253092.857143,\n'
                                            '        "bw_dev" : 627.819040,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 63030,\n'
                                            '        "iops_max" : 63617,\n'
                                            '        "iops_mean" : '
                                            '63273.214286,\n'
                                            '        "iops_stddev" : '
                                            '156.954760,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.757143,\n'
                                            '      "sys_cpu" : 38.157143,\n'
                                            '      "ctx" : 442837,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.632995,\n'
                                            '        "50" : 1.304909,\n'
                                            '        "100" : 0.052612,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'io_uring-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85266: Tue '
                                            'Oct 25 23:28:08 2022\n'
                                            '  read: IOPS=63.3k, BW=247MiB/s '
                                            '(259MB/s)(1730MiB/7001msec)\n'
                                            '    slat (nsec): min=2584, '
                                            'max=28093, avg=2753.25, '
                                            'stdev=286.50\n'
                                            '    clat (nsec): min=421, '
                                            'max=181340, avg=12481.79, '
                                            'stdev=2028.06\n'
                                            '     lat (usec): min=12, max=184, '
                                            'avg=15.24, stdev= 2.05\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[11968], 10.00th=[11968], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12096], '
                                            '40.00th=[12224], 50.00th=[12224], '
                                            '60.00th=[12224],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12352], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[27776], 99.90th=[32384], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[66048]\n'
                                            '   bw (  KiB/s): min=252120, '
                                            'max=254468, per=100.00%, '
                                            'avg=253092.86, stdev=627.82, '
                                            'samples=14\n'
                                            '   iops        : min=63030, '
                                            'max=63617, avg=63273.21, '
                                            'stdev=156.95, samples=14\n'
                                            '  lat (nsec)   : 500=0.01%, '
                                            '750=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.63%, 50=1.30%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=10.76%, '
                                            'sys=38.16%, ctx=442837, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=442866,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=247MiB/s (259MB/s), '
                                            '247MiB/s-247MiB/s '
                                            '(259MB/s-259MB/s), io=1730MiB '
                                            '(1814MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.output'),
                                  'state': {'begin': 1666726198.7879772,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_io_uring-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7874362468719482,
                                            'end': 1666726199.5754135,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_63.state')},
                       'cmd_64': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-reference_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.output'),
                                  'state': {'begin': 1666726200.562892,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7885828018188477,
                                            'end': 1666726201.3514748,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_64.state')},
                       'cmd_65': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.output'),
                                  'state': {'begin': 1666726201.3534098,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-reference_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.085954666137695,
                                            'end': 1666726212.4393644,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_65.state')},
                       'cmd_66': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733301,\n'
                                            '  "timestamp_ms" : '
                                            '1666733301938,\n'
                                            '  "time" : "Tue Oct 25 23:28:21 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1673572352,\n'
                                            '        "io_kbytes" : 1634348,\n'
                                            '        "bw_bytes" : 239047614,\n'
                                            '        "bw" : 233444,\n'
                                            '        "iops" : 58361.234109,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 408587,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2836,\n'
                                            '          "max" : 17292,\n'
                                            '          "mean" : 3074.658611,\n'
                                            '          "stddev" : 158.340253,\n'
                                            '          "N" : 408587\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 802,\n'
                                            '          "max" : 118943,\n'
                                            '          "mean" : 13512.992719,\n'
                                            '          "stddev" : '
                                            '2095.582789,\n'
                                            '          "N" : 408587,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 12864,\n'
                                            '            "5.000000" : 12992,\n'
                                            '            "10.000000" : 13120,\n'
                                            '            "20.000000" : 13120,\n'
                                            '            "30.000000" : 13248,\n'
                                            '            "40.000000" : 13248,\n'
                                            '            "50.000000" : 13248,\n'
                                            '            "60.000000" : 13248,\n'
                                            '            "70.000000" : 13248,\n'
                                            '            "80.000000" : 13376,\n'
                                            '            "90.000000" : 13376,\n'
                                            '            "95.000000" : 13504,\n'
                                            '            "99.000000" : 23680,\n'
                                            '            "99.500000" : 28800,\n'
                                            '            "99.900000" : 33536,\n'
                                            '            "99.950000" : 50432,\n'
                                            '            "99.990000" : 71168\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 14397,\n'
                                            '          "max" : 122139,\n'
                                            '          "mean" : 16587.651330,\n'
                                            '          "stddev" : '
                                            '2102.634447,\n'
                                            '          "N" : 408587\n'
                                            '        },\n'
                                            '        "bw_min" : 233168,\n'
                                            '        "bw_max" : 233976,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '233596.714286,\n'
                                            '        "bw_dev" : 275.558574,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 58292,\n'
                                            '        "iops_max" : 58494,\n'
                                            '        "iops_mean" : '
                                            '58399.000000,\n'
                                            '        "iops_stddev" : '
                                            '68.835367,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 9.100000,\n'
                                            '      "sys_cpu" : 32.528571,\n'
                                            '      "ctx" : 408588,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.564810,\n'
                                            '        "50" : 1.376696,\n'
                                            '        "100" : 0.051641,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 586804,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5712,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5712,\n'
                                            '      "util" : 99.108381\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=85307: Tue '
                                            'Oct 25 23:28:21 2022\n'
                                            '  read: IOPS=58.4k, BW=228MiB/s '
                                            '(239MB/s)(1596MiB/7001msec)\n'
                                            '    slat (nsec): min=2836, '
                                            'max=17292, avg=3074.66, '
                                            'stdev=158.34\n'
                                            '    clat (nsec): min=802, '
                                            'max=118943, avg=13512.99, '
                                            'stdev=2095.58\n'
                                            '     lat (usec): min=14, max=122, '
                                            'avg=16.59, stdev= 2.10\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[12864],  '
                                            '5.00th=[12992], 10.00th=[13120], '
                                            '20.00th=[13120],\n'
                                            '     | 30.00th=[13248], '
                                            '40.00th=[13248], 50.00th=[13248], '
                                            '60.00th=[13248],\n'
                                            '     | 70.00th=[13248], '
                                            '80.00th=[13376], 90.00th=[13376], '
                                            '95.00th=[13504],\n'
                                            '     | 99.00th=[23680], '
                                            '99.50th=[28800], 99.90th=[33536], '
                                            '99.95th=[50432],\n'
                                            '     | 99.99th=[71168]\n'
                                            '   bw (  KiB/s): min=233168, '
                                            'max=233976, per=100.00%, '
                                            'avg=233596.71, stdev=275.56, '
                                            'samples=14\n'
                                            '   iops        : min=58292, '
                                            'max=58494, avg=58399.00, '
                                            'stdev=68.84, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 10=0.01%, '
                                            '20=98.56%, 50=1.38%, 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=9.10%, '
                                            'sys=32.53%, ctx=408588, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=408587,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=228MiB/s (239MB/s), '
                                            '228MiB/s-228MiB/s '
                                            '(239MB/s-239MB/s), io=1596MiB '
                                            '(1674MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=586804/0, '
                                            'merge=0/0, ticks=5712/0, '
                                            'in_queue=5712, util=99.11%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.output'),
                                  'state': {'begin': 1666726212.4419904,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-reference_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7871720790863037,
                                            'end': 1666726213.2291625,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_66.state')},
                       'cmd_67': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-reference_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.output'),
                                  'state': {'begin': 1666726214.2225513,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7865972518920898,
                                            'end': 1666726215.0091486,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_67.state')},
                       'cmd_68': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.output'),
                                  'state': {'begin': 1666726215.0111077,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-reference_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.077730178833008,
                                            'end': 1666726226.0888379,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_68.state')},
                       'cmd_69': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733315,\n'
                                            '  "timestamp_ms" : '
                                            '1666733315587,\n'
                                            '  "time" : "Tue Oct 25 23:28:35 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1801781248,\n'
                                            '        "io_kbytes" : 1759552,\n'
                                            '        "bw_bytes" : 257360555,\n'
                                            '        "bw" : 251328,\n'
                                            '        "iops" : 62832.166833,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 439888,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2765,\n'
                                            '          "max" : 26921,\n'
                                            '          "mean" : 2932.077167,\n'
                                            '          "stddev" : 192.262685,\n'
                                            '          "N" : 439888\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 881,\n'
                                            '          "max" : 77566,\n'
                                            '          "mean" : 12358.918272,\n'
                                            '          "stddev" : '
                                            '1970.343765,\n'
                                            '          "N" : 439888,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11840,\n'
                                            '            "5.000000" : 11840,\n'
                                            '            "10.000000" : 11968,\n'
                                            '            "20.000000" : 11968,\n'
                                            '            "30.000000" : 11968,\n'
                                            '            "40.000000" : 12096,\n'
                                            '            "50.000000" : 12096,\n'
                                            '            "60.000000" : 12096,\n'
                                            '            "70.000000" : 12224,\n'
                                            '            "80.000000" : 12224,\n'
                                            '            "90.000000" : 12352,\n'
                                            '            "95.000000" : 12480,\n'
                                            '            "99.000000" : 21888,\n'
                                            '            "99.500000" : 27264,\n'
                                            '            "99.900000" : 31616,\n'
                                            '            "99.950000" : 49920,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12002,\n'
                                            '          "max" : 80582,\n'
                                            '          "mean" : 15290.995440,\n'
                                            '          "stddev" : '
                                            '1982.252033,\n'
                                            '          "N" : 439888\n'
                                            '        },\n'
                                            '        "bw_min" : 250264,\n'
                                            '        "bw_max" : 252440,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '251431.928571,\n'
                                            '        "bw_dev" : 631.720540,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 62566,\n'
                                            '        "iops_max" : 63110,\n'
                                            '        "iops_mean" : '
                                            '62857.928571,\n'
                                            '        "iops_stddev" : '
                                            '157.898928,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 11.057143,\n'
                                            '      "sys_cpu" : 37.400000,\n'
                                            '      "ctx" : 439880,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.638062,\n'
                                            '        "50" : 1.307151,\n'
                                            '        "100" : 0.047057,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 631256,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 5943,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 5943,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=85345: Tue '
                                            'Oct 25 23:28:35 2022\n'
                                            '  read: IOPS=62.8k, BW=245MiB/s '
                                            '(257MB/s)(1718MiB/7001msec)\n'
                                            '    slat (nsec): min=2765, '
                                            'max=26921, avg=2932.08, '
                                            'stdev=192.26\n'
                                            '    clat (nsec): min=881, '
                                            'max=77566, avg=12358.92, '
                                            'stdev=1970.34\n'
                                            '     lat (nsec): min=12002, '
                                            'max=80582, avg=15291.00, '
                                            'stdev=1982.25\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11840],  '
                                            '5.00th=[11840], 10.00th=[11968], '
                                            '20.00th=[11968],\n'
                                            '     | 30.00th=[11968], '
                                            '40.00th=[12096], 50.00th=[12096], '
                                            '60.00th=[12096],\n'
                                            '     | 70.00th=[12224], '
                                            '80.00th=[12224], 90.00th=[12352], '
                                            '95.00th=[12480],\n'
                                            '     | 99.00th=[21888], '
                                            '99.50th=[27264], 99.90th=[31616], '
                                            '99.95th=[49920],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=250264, '
                                            'max=252440, per=100.00%, '
                                            'avg=251431.93, stdev=631.72, '
                                            'samples=14\n'
                                            '   iops        : min=62566, '
                                            'max=63110, avg=62857.93, '
                                            'stdev=157.90, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=0.01%, 20=98.64%, 50=1.31%, '
                                            '100=0.05%\n'
                                            '  cpu          : usr=11.06%, '
                                            'sys=37.40%, ctx=439880, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=439888,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=245MiB/s (257MB/s), '
                                            '245MiB/s-245MiB/s '
                                            '(257MB/s-257MB/s), io=1718MiB '
                                            '(1802MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=631256/0, '
                                            'merge=0/0, ticks=5943/0, '
                                            'in_queue=5943, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.output'),
                                  'state': {'begin': 1666726226.0914607,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-reference_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7961251735687256,
                                            'end': 1666726226.8875859,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_69.state')},
                       'cmd_70': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-reference_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.output'),
                                  'state': {'begin': 1666726227.883852,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7791063785552979,
                                            'end': 1666726228.6629584,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_70.state')},
                       'cmd_71': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.output'),
                                  'state': {'begin': 1666726228.665182,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-reference" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="libaio" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-reference_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.08476448059082,
                                            'end': 1666726239.7499466,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_71.state')},
                       'cmd_72': {'output': 'libaio-reference: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=libaio, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733329,\n'
                                            '  "timestamp_ms" : '
                                            '1666733329246,\n'
                                            '  "time" : "Tue Oct 25 23:28:49 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-reference",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-reference",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : "libaio",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1831936000,\n'
                                            '        "io_kbytes" : 1789000,\n'
                                            '        "bw_bytes" : 261667761,\n'
                                            '        "bw" : 255534,\n'
                                            '        "iops" : 63883.588059,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 447249,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2605,\n'
                                            '          "max" : 37320,\n'
                                            '          "mean" : 2851.803557,\n'
                                            '          "stddev" : 243.670282,\n'
                                            '          "N" : 447250\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 872,\n'
                                            '          "max" : 136185,\n'
                                            '          "mean" : 12256.182628,\n'
                                            '          "stddev" : '
                                            '1988.047396,\n'
                                            '          "N" : 447249,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11712,\n'
                                            '            "5.000000" : 11712,\n'
                                            '            "10.000000" : 11840,\n'
                                            '            "20.000000" : 11840,\n'
                                            '            "30.000000" : 11840,\n'
                                            '            "40.000000" : 11968,\n'
                                            '            "50.000000" : 11968,\n'
                                            '            "60.000000" : 12096,\n'
                                            '            "70.000000" : 12096,\n'
                                            '            "80.000000" : 12096,\n'
                                            '            "90.000000" : 12224,\n'
                                            '            "95.000000" : 12352,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 26752,\n'
                                            '            "99.900000" : 31616,\n'
                                            '            "99.950000" : 49408,\n'
                                            '            "99.990000" : 63744\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12163,\n'
                                            '          "max" : 138971,\n'
                                            '          "mean" : 15107.986178,\n'
                                            '          "stddev" : '
                                            '2008.944842,\n'
                                            '          "N" : 447249\n'
                                            '        },\n'
                                            '        "bw_min" : 254784,\n'
                                            '        "bw_max" : 257032,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '255632.357143,\n'
                                            '        "bw_dev" : 623.778265,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 63696,\n'
                                            '        "iops_max" : 64258,\n'
                                            '        "iops_mean" : '
                                            '63908.000000,\n'
                                            '        "iops_stddev" : '
                                            '155.974850,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 10.242857,\n'
                                            '      "sys_cpu" : 37.271429,\n'
                                            '      "ctx" : 447233,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.553155,\n'
                                            '        "50" : 1.395867,\n'
                                            '        "100" : 0.044047,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ],\n'
                                            '  "disk_util" : [\n'
                                            '    {\n'
                                            '      "name" : "nvme0n1",\n'
                                            '      "read_ios" : 640196,\n'
                                            '      "write_ios" : 0,\n'
                                            '      "read_merges" : 0,\n'
                                            '      "write_merges" : 0,\n'
                                            '      "read_ticks" : 6020,\n'
                                            '      "write_ticks" : 0,\n'
                                            '      "in_queue" : 6020,\n'
                                            '      "util" : 99.088748\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-reference: (groupid=0, '
                                            'jobs=1): err= 0: pid=85383: Tue '
                                            'Oct 25 23:28:49 2022\n'
                                            '  read: IOPS=63.9k, BW=250MiB/s '
                                            '(262MB/s)(1747MiB/7001msec)\n'
                                            '    slat (nsec): min=2605, '
                                            'max=37320, avg=2851.80, '
                                            'stdev=243.67\n'
                                            '    clat (nsec): min=872, '
                                            'max=136185, avg=12256.18, '
                                            'stdev=1988.05\n'
                                            '     lat (usec): min=12, max=138, '
                                            'avg=15.11, stdev= 2.01\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11712],  '
                                            '5.00th=[11712], 10.00th=[11840], '
                                            '20.00th=[11840],\n'
                                            '     | 30.00th=[11840], '
                                            '40.00th=[11968], 50.00th=[11968], '
                                            '60.00th=[12096],\n'
                                            '     | 70.00th=[12096], '
                                            '80.00th=[12096], 90.00th=[12224], '
                                            '95.00th=[12352],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[26752], 99.90th=[31616], '
                                            '99.95th=[49408],\n'
                                            '     | 99.99th=[63744]\n'
                                            '   bw (  KiB/s): min=254784, '
                                            'max=257032, per=100.00%, '
                                            'avg=255632.36, stdev=623.78, '
                                            'samples=14\n'
                                            '   iops        : min=63696, '
                                            'max=64258, avg=63908.00, '
                                            'stdev=155.97, samples=14\n'
                                            '  lat (nsec)   : 1000=0.01%\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=0.01%, 20=98.55%, '
                                            '50=1.40%\n'
                                            '  lat (usec)   : 100=0.04%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=10.24%, '
                                            'sys=37.27%, ctx=447233, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=447249,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=250MiB/s (262MB/s), '
                                            '250MiB/s-250MiB/s '
                                            '(262MB/s-262MB/s), io=1747MiB '
                                            '(1832MB), run=7001-7001msec\n'
                                            '\n'
                                            'Disk stats (read/write):\n'
                                            '  nvme0n1: ios=640196/0, '
                                            'merge=0/0, ticks=6020/0, '
                                            'in_queue=6020, util=99.09%\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.output'),
                                  'state': {'begin': 1666726239.75269,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-reference_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7971940040588379,
                                            'end': 1666726240.549884,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_72.state')},
                       'cmd_73': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.output'),
                                  'state': {'begin': 1666726242.5884023,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7865762710571289,
                                            'end': 1666726243.3749785,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_73.state')},
                       'cmd_74': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.output'),
                                  'state': {'begin': 1666726243.3770688,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.566319227218628,
                                            'end': 1666726254.943388,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_74.state')},
                       'cmd_75': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733344,\n'
                                            '  "timestamp_ms" : '
                                            '1666733344307,\n'
                                            '  "time" : "Tue Oct 25 23:29:04 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2908217344,\n'
                                            '        "io_kbytes" : 2840056,\n'
                                            '        "bw_bytes" : 415400277,\n'
                                            '        "bw" : 405664,\n'
                                            '        "iops" : 101416.083417,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 710014,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1562,\n'
                                            '          "max" : 34896,\n'
                                            '          "mean" : 1634.305671,\n'
                                            '          "stddev" : 120.389758,\n'
                                            '          "N" : 710014\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 221,\n'
                                            '          "max" : 113843,\n'
                                            '          "mean" : 7995.203314,\n'
                                            '          "stddev" : '
                                            '1932.985662,\n'
                                            '          "N" : 710014,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7520,\n'
                                            '            "5.000000" : 7520,\n'
                                            '            "10.000000" : 7520,\n'
                                            '            "20.000000" : 7584,\n'
                                            '            "30.000000" : 7584,\n'
                                            '            "40.000000" : 7648,\n'
                                            '            "50.000000" : 7712,\n'
                                            '            "60.000000" : 7712,\n'
                                            '            "70.000000" : 7776,\n'
                                            '            "80.000000" : 7840,\n'
                                            '            "90.000000" : 7840,\n'
                                            '            "95.000000" : 8384,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 19840,\n'
                                            '            "99.900000" : 23680,\n'
                                            '            "99.950000" : 40704,\n'
                                            '            "99.990000" : 65280\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9027,\n'
                                            '          "max" : 115476,\n'
                                            '          "mean" : 9629.508986,\n'
                                            '          "stddev" : '
                                            '1936.932387,\n'
                                            '          "N" : 710014\n'
                                            '        },\n'
                                            '        "bw_min" : 404904,\n'
                                            '        "bw_max" : 407032,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '405763.785714,\n'
                                            '        "bw_dev" : 726.004893,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 101226,\n'
                                            '        "iops_max" : 101758,\n'
                                            '        "iops_mean" : '
                                            '101440.928571,\n'
                                            '        "iops_stddev" : '
                                            '181.477044,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 85.300000,\n'
                                            '      "sys_cpu" : 14.671429,\n'
                                            '      "ctx" : 18,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.102471,\n'
                                            '        "20" : 3.421904,\n'
                                            '        "50" : 0.436611,\n'
                                            '        "100" : 0.037605,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=85432: Tue '
                                            'Oct 25 23:29:04 2022\n'
                                            '  read: IOPS=101k, BW=396MiB/s '
                                            '(415MB/s)(2773MiB/7001msec)\n'
                                            '    slat (nsec): min=1562, '
                                            'max=34896, avg=1634.31, '
                                            'stdev=120.39\n'
                                            '    clat (nsec): min=221, '
                                            'max=113843, avg=7995.20, '
                                            'stdev=1932.99\n'
                                            '     lat (usec): min=9, max=115, '
                                            'avg= 9.63, stdev= 1.94\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7520],  5.00th=[ '
                                            '7520], 10.00th=[ 7520], 20.00th=[ '
                                            '7584],\n'
                                            '     | 30.00th=[ 7584], 40.00th=[ '
                                            '7648], 50.00th=[ 7712], 60.00th=[ '
                                            '7712],\n'
                                            '     | 70.00th=[ 7776], 80.00th=[ '
                                            '7840], 90.00th=[ 7840], 95.00th=[ '
                                            '8384],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[19840], 99.90th=[23680], '
                                            '99.95th=[40704],\n'
                                            '     | 99.99th=[65280]\n'
                                            '   bw (  KiB/s): min=404904, '
                                            'max=407032, per=100.00%, '
                                            'avg=405763.79, stdev=726.00, '
                                            'samples=14\n'
                                            '   iops        : min=101226, '
                                            'max=101758, avg=101440.93, '
                                            'stdev=181.48, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=96.10%, '
                                            '20=3.42%, 50=0.44%, 100=0.04%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=85.30%, '
                                            'sys=14.67%, ctx=18, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=710014,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=396MiB/s (415MB/s), '
                                            '396MiB/s-396MiB/s '
                                            '(415MB/s-415MB/s), io=2773MiB '
                                            '(2908MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.output'),
                                  'state': {'begin': 1666726254.9463606,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7987728118896484,
                                            'end': 1666726255.7451334,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_75.state')},
                       'cmd_76': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.output'),
                                  'state': {'begin': 1666726257.8994217,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7967190742492676,
                                            'end': 1666726258.6961408,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_76.state')},
                       'cmd_77': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.output'),
                                  'state': {'begin': 1666726258.698414,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.520707607269287,
                                            'end': 1666726270.2191217,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_77.state')},
                       'cmd_78': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733359,\n'
                                            '  "timestamp_ms" : '
                                            '1666733359618,\n'
                                            '  "time" : "Tue Oct 25 23:29:19 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2864140288,\n'
                                            '        "io_kbytes" : 2797012,\n'
                                            '        "bw_bytes" : 409104454,\n'
                                            '        "bw" : 399516,\n'
                                            '        "iops" : 99879.017283,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 699253,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1643,\n'
                                            '          "max" : 27060,\n'
                                            '          "mean" : 1697.425543,\n'
                                            '          "stddev" : 169.441086,\n'
                                            '          "N" : 699253\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 210,\n'
                                            '          "max" : 77946,\n'
                                            '          "mean" : 8068.355484,\n'
                                            '          "stddev" : '
                                            '1872.173034,\n'
                                            '          "N" : 699253,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7648,\n'
                                            '            "5.000000" : 7712,\n'
                                            '            "10.000000" : 7712,\n'
                                            '            "20.000000" : 7712,\n'
                                            '            "30.000000" : 7712,\n'
                                            '            "40.000000" : 7712,\n'
                                            '            "50.000000" : 7776,\n'
                                            '            "60.000000" : 7776,\n'
                                            '            "70.000000" : 7776,\n'
                                            '            "80.000000" : 7776,\n'
                                            '            "90.000000" : 7904,\n'
                                            '            "95.000000" : 8512,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 19328,\n'
                                            '            "99.900000" : 23936,\n'
                                            '            "99.950000" : 36608,\n'
                                            '            "99.990000" : 64768\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9247,\n'
                                            '          "max" : 79629,\n'
                                            '          "mean" : 9765.781026,\n'
                                            '          "stddev" : '
                                            '1879.123838,\n'
                                            '          "N" : 699253\n'
                                            '        },\n'
                                            '        "bw_min" : 398744,\n'
                                            '        "bw_max" : 400280,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '399610.785714,\n'
                                            '        "bw_dev" : 354.371856,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 99686,\n'
                                            '        "iops_max" : 100070,\n'
                                            '        "iops_mean" : '
                                            '99902.642857,\n'
                                            '        "iops_stddev" : '
                                            '88.578177,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 85.242857,\n'
                                            '      "sys_cpu" : 14.728571,\n'
                                            '      "ctx" : 12,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.071522,\n'
                                            '        "20" : 3.639741,\n'
                                            '        "50" : 0.253127,\n'
                                            '        "100" : 0.032034,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=85481: Tue '
                                            'Oct 25 23:29:19 2022\n'
                                            '  read: IOPS=99.9k, BW=390MiB/s '
                                            '(409MB/s)(2731MiB/7001msec)\n'
                                            '    slat (nsec): min=1643, '
                                            'max=27060, avg=1697.43, '
                                            'stdev=169.44\n'
                                            '    clat (nsec): min=210, '
                                            'max=77946, avg=8068.36, '
                                            'stdev=1872.17\n'
                                            '     lat (nsec): min=9247, '
                                            'max=79629, avg=9765.78, '
                                            'stdev=1879.12\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7648],  5.00th=[ '
                                            '7712], 10.00th=[ 7712], 20.00th=[ '
                                            '7712],\n'
                                            '     | 30.00th=[ 7712], 40.00th=[ '
                                            '7712], 50.00th=[ 7776], 60.00th=[ '
                                            '7776],\n'
                                            '     | 70.00th=[ 7776], 80.00th=[ '
                                            '7776], 90.00th=[ 7904], 95.00th=[ '
                                            '8512],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[19328], 99.90th=[23936], '
                                            '99.95th=[36608],\n'
                                            '     | 99.99th=[64768]\n'
                                            '   bw (  KiB/s): min=398744, '
                                            'max=400280, per=100.00%, '
                                            'avg=399610.79, stdev=354.37, '
                                            'samples=14\n'
                                            '   iops        : min=99686, '
                                            'max=100070, avg=99902.64, '
                                            'stdev=88.58, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 10=96.07%, '
                                            '20=3.64%, 50=0.25%, 100=0.03%\n'
                                            '  cpu          : usr=85.24%, '
                                            'sys=14.73%, ctx=12, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=699253,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=390MiB/s (409MB/s), '
                                            '390MiB/s-390MiB/s '
                                            '(409MB/s-409MB/s), io=2731MiB '
                                            '(2864MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.output'),
                                  'state': {'begin': 1666726270.2221115,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7994446754455566,
                                            'end': 1666726271.0215561,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_78.state')},
                       'cmd_79': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.output'),
                                  'state': {'begin': 1666726273.0808809,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7926232814788818,
                                            'end': 1666726273.8735042,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_79.state')},
                       'cmd_80': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.output'),
                                  'state': {'begin': 1666726273.8753877,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_aio" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_aio_foo.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.423839807510376,
                                            'end': 1666726285.2992275,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_80.state')},
                       'cmd_81': {'output': 'libaio-bdev_aio: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733374,\n'
                                            '  "timestamp_ms" : '
                                            '1666733374688,\n'
                                            '  "time" : "Tue Oct 25 23:29:34 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_aio",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_aio",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_aio_foo.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2917703680,\n'
                                            '        "io_kbytes" : 2849320,\n'
                                            '        "bw_bytes" : 416814811,\n'
                                            '        "bw" : 407045,\n'
                                            '        "iops" : 101761.428571,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 712330,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1533,\n'
                                            '          "max" : 13756,\n'
                                            '          "mean" : 1593.976842,\n'
                                            '          "stddev" : 115.555755,\n'
                                            '          "N" : 712330\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 221,\n'
                                            '          "max" : 75782,\n'
                                            '          "mean" : 7996.381760,\n'
                                            '          "stddev" : '
                                            '1896.986393,\n'
                                            '          "N" : 712330,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7520,\n'
                                            '            "5.000000" : 7520,\n'
                                            '            "10.000000" : 7520,\n'
                                            '            "20.000000" : 7584,\n'
                                            '            "30.000000" : 7584,\n'
                                            '            "40.000000" : 7648,\n'
                                            '            "50.000000" : 7712,\n'
                                            '            "60.000000" : 7712,\n'
                                            '            "70.000000" : 7776,\n'
                                            '            "80.000000" : 7840,\n'
                                            '            "90.000000" : 7840,\n'
                                            '            "95.000000" : 8256,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 20096,\n'
                                            '            "99.900000" : 22656,\n'
                                            '            "99.950000" : 32384,\n'
                                            '            "99.990000" : 65280\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9007,\n'
                                            '          "max" : 77365,\n'
                                            '          "mean" : 9590.358602,\n'
                                            '          "stddev" : '
                                            '1900.929895,\n'
                                            '          "N" : 712330\n'
                                            '        },\n'
                                            '        "bw_min" : 406344,\n'
                                            '        "bw_max" : 407800,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '407090.071429,\n'
                                            '        "bw_dev" : 420.417094,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 101586,\n'
                                            '        "iops_max" : 101950,\n'
                                            '        "iops_mean" : '
                                            '101772.500000,\n'
                                            '        "iops_stddev" : '
                                            '105.118797,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 85.871429,\n'
                                            '      "sys_cpu" : 14.100000,\n'
                                            '      "ctx" : 10,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 96.124268,\n'
                                            '        "20" : 3.367821,\n'
                                            '        "50" : 0.472955,\n'
                                            '        "100" : 0.033692,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_aio: (groupid=0, '
                                            'jobs=1): err= 0: pid=85531: Tue '
                                            'Oct 25 23:29:34 2022\n'
                                            '  read: IOPS=102k, BW=398MiB/s '
                                            '(417MB/s)(2783MiB/7000msec)\n'
                                            '    slat (nsec): min=1533, '
                                            'max=13756, avg=1593.98, '
                                            'stdev=115.56\n'
                                            '    clat (nsec): min=221, '
                                            'max=75782, avg=7996.38, '
                                            'stdev=1896.99\n'
                                            '     lat (nsec): min=9007, '
                                            'max=77365, avg=9590.36, '
                                            'stdev=1900.93\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7520],  5.00th=[ '
                                            '7520], 10.00th=[ 7520], 20.00th=[ '
                                            '7584],\n'
                                            '     | 30.00th=[ 7584], 40.00th=[ '
                                            '7648], 50.00th=[ 7712], 60.00th=[ '
                                            '7712],\n'
                                            '     | 70.00th=[ 7776], 80.00th=[ '
                                            '7840], 90.00th=[ 7840], 95.00th=[ '
                                            '8256],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[20096], 99.90th=[22656], '
                                            '99.95th=[32384],\n'
                                            '     | 99.99th=[65280]\n'
                                            '   bw (  KiB/s): min=406344, '
                                            'max=407800, per=100.00%, '
                                            'avg=407090.07, stdev=420.42, '
                                            'samples=14\n'
                                            '   iops        : min=101586, '
                                            'max=101950, avg=101772.50, '
                                            'stdev=105.12, samples=14\n'
                                            '  lat (nsec)   : 250=0.01%, '
                                            '500=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=96.12%, 20=3.37%, 50=0.47%, '
                                            '100=0.03%\n'
                                            '  cpu          : usr=85.87%, '
                                            'sys=14.10%, ctx=10, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=712330,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=398MiB/s (417MB/s), '
                                            '398MiB/s-398MiB/s '
                                            '(417MB/s-417MB/s), io=2783MiB '
                                            '(2918MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.output'),
                                  'state': {'begin': 1666726285.3017569,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8237886428833008,
                                            'end': 1666726286.1255455,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_81.state')},
                       'cmd_82': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.output'),
                                  'state': {'begin': 1666726288.2137384,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7974402904510498,
                                            'end': 1666726289.0111787,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_82.state')},
                       'cmd_83': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.output'),
                                  'state': {'begin': 1666726289.013351,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.530867338180542,
                                            'end': 1666726300.5442183,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_83.state')},
                       'cmd_84': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733389,\n'
                                            '  "timestamp_ms" : '
                                            '1666733389939,\n'
                                            '  "time" : "Tue Oct 25 23:29:49 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2542465024,\n'
                                            '        "io_kbytes" : 2482876,\n'
                                            '        "bw_bytes" : 363157409,\n'
                                            '        "bw" : 354645,\n'
                                            '        "iops" : 88661.476932,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 620719,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1783,\n'
                                            '          "max" : 14928,\n'
                                            '          "mean" : 1926.784324,\n'
                                            '          "stddev" : 147.911807,\n'
                                            '          "N" : 620719\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 611,\n'
                                            '          "max" : 95610,\n'
                                            '          "mean" : 9103.494547,\n'
                                            '          "stddev" : '
                                            '2144.724722,\n'
                                            '          "N" : 620719,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 8512,\n'
                                            '            "5.000000" : 8640,\n'
                                            '            "10.000000" : 8640,\n'
                                            '            "20.000000" : 8640,\n'
                                            '            "30.000000" : 8768,\n'
                                            '            "40.000000" : 8768,\n'
                                            '            "50.000000" : 8768,\n'
                                            '            "60.000000" : 8768,\n'
                                            '            "70.000000" : 8768,\n'
                                            '            "80.000000" : 8896,\n'
                                            '            "90.000000" : 9024,\n'
                                            '            "95.000000" : 9536,\n'
                                            '            "99.000000" : 19328,\n'
                                            '            "99.500000" : 24960,\n'
                                            '            "99.900000" : 26240,\n'
                                            '            "99.950000" : 41728,\n'
                                            '            "99.990000" : 65280\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9959,\n'
                                            '          "max" : 97583,\n'
                                            '          "mean" : 11030.278872,\n'
                                            '          "stddev" : '
                                            '2148.473123,\n'
                                            '          "N" : 620719\n'
                                            '        },\n'
                                            '        "bw_min" : 353864,\n'
                                            '        "bw_max" : 355863,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '354830.928571,\n'
                                            '        "bw_dev" : 468.121198,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 88466,\n'
                                            '        "iops_max" : 88965,\n'
                                            '        "iops_mean" : '
                                            '88707.642857,\n'
                                            '        "iops_stddev" : '
                                            '116.917201,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 34.157143,\n'
                                            '      "sys_cpu" : 65.814286,\n'
                                            '      "ctx" : 19,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 95.638445,\n'
                                            '        "20" : 3.521722,\n'
                                            '        "50" : 0.793274,\n'
                                            '        "100" : 0.039309,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85581: Tue '
                                            'Oct 25 23:29:49 2022\n'
                                            '  read: IOPS=88.7k, BW=346MiB/s '
                                            '(363MB/s)(2425MiB/7001msec)\n'
                                            '    slat (nsec): min=1783, '
                                            'max=14928, avg=1926.78, '
                                            'stdev=147.91\n'
                                            '    clat (nsec): min=611, '
                                            'max=95610, avg=9103.49, '
                                            'stdev=2144.72\n'
                                            '     lat (nsec): min=9959, '
                                            'max=97583, avg=11030.28, '
                                            'stdev=2148.47\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 8512],  5.00th=[ '
                                            '8640], 10.00th=[ 8640], 20.00th=[ '
                                            '8640],\n'
                                            '     | 30.00th=[ 8768], 40.00th=[ '
                                            '8768], 50.00th=[ 8768], 60.00th=[ '
                                            '8768],\n'
                                            '     | 70.00th=[ 8768], 80.00th=[ '
                                            '8896], 90.00th=[ 9024], 95.00th=[ '
                                            '9536],\n'
                                            '     | 99.00th=[19328], '
                                            '99.50th=[24960], 99.90th=[26240], '
                                            '99.95th=[41728],\n'
                                            '     | 99.99th=[65280]\n'
                                            '   bw (  KiB/s): min=353864, '
                                            'max=355863, per=100.00%, '
                                            'avg=354830.93, stdev=468.12, '
                                            'samples=14\n'
                                            '   iops        : min=88466, '
                                            'max=88965, avg=88707.64, '
                                            'stdev=116.92, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=95.64%, 20=3.52%, 50=0.79%, '
                                            '100=0.04%\n'
                                            '  cpu          : usr=34.16%, '
                                            'sys=65.81%, ctx=19, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=620719,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=346MiB/s (363MB/s), '
                                            '346MiB/s-346MiB/s '
                                            '(363MB/s-363MB/s), io=2425MiB '
                                            '(2542MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.output'),
                                  'state': {'begin': 1666726300.5469594,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7896029949188232,
                                            'end': 1666726301.3365624,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_84.state')},
                       'cmd_85': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.output'),
                                  'state': {'begin': 1666726303.3699868,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7868921756744385,
                                            'end': 1666726304.156879,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_85.state')},
                       'cmd_86': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.output'),
                                  'state': {'begin': 1666726304.1589577,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.434263229370117,
                                            'end': 1666726315.593221,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_86.state')},
                       'cmd_87': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733404,\n'
                                            '  "timestamp_ms" : '
                                            '1666733404980,\n'
                                            '  "time" : "Tue Oct 25 23:30:04 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2823823360,\n'
                                            '        "io_kbytes" : 2757640,\n'
                                            '        "bw_bytes" : 403403337,\n'
                                            '        "bw" : 393948,\n'
                                            '        "iops" : 98487.142857,\n'
                                            '        "runtime" : 7000,\n'
                                            '        "total_ios" : 689410,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1582,\n'
                                            '          "max" : 36869,\n'
                                            '          "mean" : 1652.357284,\n'
                                            '          "stddev" : 116.694315,\n'
                                            '          "N" : 689410\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 651,\n'
                                            '          "max" : 112781,\n'
                                            '          "mean" : 8251.688869,\n'
                                            '          "stddev" : '
                                            '1885.218026,\n'
                                            '          "N" : 689410,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7776,\n'
                                            '            "5.000000" : 7776,\n'
                                            '            "10.000000" : 7776,\n'
                                            '            "20.000000" : 7840,\n'
                                            '            "30.000000" : 7840,\n'
                                            '            "40.000000" : 7904,\n'
                                            '            "50.000000" : 7904,\n'
                                            '            "60.000000" : 7904,\n'
                                            '            "70.000000" : 8032,\n'
                                            '            "80.000000" : 8160,\n'
                                            '            "90.000000" : 8256,\n'
                                            '            "95.000000" : 8640,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 19328,\n'
                                            '            "99.900000" : 22656,\n'
                                            '            "99.950000" : 41216,\n'
                                            '            "99.990000" : 67072\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9198,\n'
                                            '          "max" : 114405,\n'
                                            '          "mean" : 9904.046153,\n'
                                            '          "stddev" : '
                                            '1889.202305,\n'
                                            '          "N" : 689410\n'
                                            '        },\n'
                                            '        "bw_min" : 393288,\n'
                                            '        "bw_max" : 394592,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '393990.571429,\n'
                                            '        "bw_dev" : 426.285876,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 98322,\n'
                                            '        "iops_max" : 98648,\n'
                                            '        "iops_mean" : '
                                            '98497.642857,\n'
                                            '        "iops_stddev" : '
                                            '106.571469,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 33.642857,\n'
                                            '      "sys_cpu" : 66.328571,\n'
                                            '      "ctx" : 17,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 96.026167,\n'
                                            '        "20" : 3.730146,\n'
                                            '        "50" : 0.206408,\n'
                                            '        "100" : 0.035683,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85630: Tue '
                                            'Oct 25 23:30:04 2022\n'
                                            '  read: IOPS=98.5k, BW=385MiB/s '
                                            '(403MB/s)(2693MiB/7000msec)\n'
                                            '    slat (nsec): min=1582, '
                                            'max=36869, avg=1652.36, '
                                            'stdev=116.69\n'
                                            '    clat (nsec): min=651, '
                                            'max=112781, avg=8251.69, '
                                            'stdev=1885.22\n'
                                            '     lat (usec): min=9, max=114, '
                                            'avg= 9.90, stdev= 1.89\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7776],  5.00th=[ '
                                            '7776], 10.00th=[ 7776], 20.00th=[ '
                                            '7840],\n'
                                            '     | 30.00th=[ 7840], 40.00th=[ '
                                            '7904], 50.00th=[ 7904], 60.00th=[ '
                                            '7904],\n'
                                            '     | 70.00th=[ 8032], 80.00th=[ '
                                            '8160], 90.00th=[ 8256], 95.00th=[ '
                                            '8640],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[19328], 99.90th=[22656], '
                                            '99.95th=[41216],\n'
                                            '     | 99.99th=[67072]\n'
                                            '   bw (  KiB/s): min=393288, '
                                            'max=394592, per=100.00%, '
                                            'avg=393990.57, stdev=426.29, '
                                            'samples=14\n'
                                            '   iops        : min=98322, '
                                            'max=98648, avg=98497.64, '
                                            'stdev=106.57, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%\n'
                                            '  lat (usec)   : 4=0.01%, '
                                            '10=96.03%, 20=3.73%, 50=0.21%, '
                                            '100=0.04%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=33.64%, '
                                            'sys=66.33%, ctx=17, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=689410,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=385MiB/s (403MB/s), '
                                            '385MiB/s-385MiB/s '
                                            '(403MB/s-403MB/s), io=2693MiB '
                                            '(2824MB), run=7000-7000msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.output'),
                                  'state': {'begin': 1666726315.5958962,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8652126789093018,
                                            'end': 1666726316.461109,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_87.state')},
                       'cmd_88': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.output'),
                                  'state': {'begin': 1666726318.6691997,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8573663234710693,
                                            'end': 1666726319.526566,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_88.state')},
                       'cmd_89': {'output': 'TELEMETRY: No legacy callbacks, '
                                            'legacy socket not created\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.output'),
                                  'state': {'begin': 1666726319.5289133,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-bdev_xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="spdk_bdev" '
                                                   '--spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" '
                                                   '--filename="Nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.511648178100586,
                                            'end': 1666726331.0405614,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_89.state')},
                       'cmd_90': {'output': 'libaio-bdev_xnvme: (g=0): '
                                            'rw=randread, bs=(R) 4096B-4096B, '
                                            '(W) 4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=spdk_bdev, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733420,\n'
                                            '  "timestamp_ms" : '
                                            '1666733420440,\n'
                                            '  "time" : "Tue Oct 25 23:30:20 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : '
                                            '"libaio-bdev_xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"spdk_bdev",\n'
                                            '        "spdk_conf" : '
                                            '"/tmp/spdk_bdev_xnvme_libaio.conf",\n'
                                            '        "filename" : "Nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 2837221376,\n'
                                            '        "io_kbytes" : 2770724,\n'
                                            '        "bw_bytes" : 405259445,\n'
                                            '        "bw" : 395761,\n'
                                            '        "iops" : 98940.294244,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 692681,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 1563,\n'
                                            '          "max" : 52007,\n'
                                            '          "mean" : 1627.296386,\n'
                                            '          "stddev" : 124.623637,\n'
                                            '          "N" : 692681\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 631,\n'
                                            '          "max" : 93886,\n'
                                            '          "mean" : 8236.372760,\n'
                                            '          "stddev" : '
                                            '1870.436580,\n'
                                            '          "N" : 692681,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 7584,\n'
                                            '            "5.000000" : 7648,\n'
                                            '            "10.000000" : 7840,\n'
                                            '            "20.000000" : 7904,\n'
                                            '            "30.000000" : 7904,\n'
                                            '            "40.000000" : 7904,\n'
                                            '            "50.000000" : 7968,\n'
                                            '            "60.000000" : 7968,\n'
                                            '            "70.000000" : 7968,\n'
                                            '            "80.000000" : 7968,\n'
                                            '            "90.000000" : 8032,\n'
                                            '            "95.000000" : 8640,\n'
                                            '            "99.000000" : 17280,\n'
                                            '            "99.500000" : 19328,\n'
                                            '            "99.900000" : 22656,\n'
                                            '            "99.950000" : 40704,\n'
                                            '            "99.990000" : 66048\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 9167,\n'
                                            '          "max" : 95499,\n'
                                            '          "mean" : 9863.669146,\n'
                                            '          "stddev" : '
                                            '1874.856856,\n'
                                            '          "N" : 692681\n'
                                            '        },\n'
                                            '        "bw_min" : 395336,\n'
                                            '        "bw_max" : 396672,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '395852.500000,\n'
                                            '        "bw_dev" : 386.086230,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 98834,\n'
                                            '        "iops_max" : 99168,\n'
                                            '        "iops_mean" : '
                                            '98963.071429,\n'
                                            '        "iops_stddev" : '
                                            '96.554858,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 28.842857,\n'
                                            '      "sys_cpu" : 71.128571,\n'
                                            '      "ctx" : 44,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.010000,\n'
                                            '        "1000" : 0.010000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 96.036992,\n'
                                            '        "20" : 3.718595,\n'
                                            '        "50" : 0.209476,\n'
                                            '        "100" : 0.033926,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-bdev_xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85679: Tue '
                                            'Oct 25 23:30:20 2022\n'
                                            '  read: IOPS=98.9k, BW=386MiB/s '
                                            '(405MB/s)(2706MiB/7001msec)\n'
                                            '    slat (nsec): min=1563, '
                                            'max=52007, avg=1627.30, '
                                            'stdev=124.62\n'
                                            '    clat (nsec): min=631, '
                                            'max=93886, avg=8236.37, '
                                            'stdev=1870.44\n'
                                            '     lat (nsec): min=9167, '
                                            'max=95499, avg=9863.67, '
                                            'stdev=1874.86\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[ 7584],  5.00th=[ '
                                            '7648], 10.00th=[ 7840], 20.00th=[ '
                                            '7904],\n'
                                            '     | 30.00th=[ 7904], 40.00th=[ '
                                            '7904], 50.00th=[ 7968], 60.00th=[ '
                                            '7968],\n'
                                            '     | 70.00th=[ 7968], 80.00th=[ '
                                            '7968], 90.00th=[ 8032], 95.00th=[ '
                                            '8640],\n'
                                            '     | 99.00th=[17280], '
                                            '99.50th=[19328], 99.90th=[22656], '
                                            '99.95th=[40704],\n'
                                            '     | 99.99th=[66048]\n'
                                            '   bw (  KiB/s): min=395336, '
                                            'max=396672, per=100.00%, '
                                            'avg=395852.50, stdev=386.09, '
                                            'samples=14\n'
                                            '   iops        : min=98834, '
                                            'max=99168, avg=98963.07, '
                                            'stdev=96.55, samples=14\n'
                                            '  lat (nsec)   : 750=0.01%, '
                                            '1000=0.01%\n'
                                            '  lat (usec)   : 10=96.04%, '
                                            '20=3.72%, 50=0.21%, 100=0.03%\n'
                                            '  cpu          : usr=28.84%, '
                                            'sys=71.13%, ctx=44, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=692681,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=386MiB/s (405MB/s), '
                                            '386MiB/s-386MiB/s '
                                            '(405MB/s-405MB/s), io=2706MiB '
                                            '(2837MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.output'),
                                  'state': {'begin': 1666726331.043958,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7904596328735352,
                                            'end': 1666726331.8344176,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_90.state')},
                       'cmd_91': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-xnvme_0.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.output'),
                                  'state': {'begin': 1666726332.8733482,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7861638069152832,
                                            'end': 1666726333.659512,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_91.state')},
                       'cmd_92': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.output'),
                                  'state': {'begin': 1666726333.6615798,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-xnvme_0.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.092099666595459,
                                            'end': 1666726344.7536795,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_92.state')},
                       'cmd_93': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 4096B-4096B, (W) '
                                            '4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733434,\n'
                                            '  "timestamp_ms" : '
                                            '1666733434249,\n'
                                            '  "time" : "Tue Oct 25 23:30:34 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1678741504,\n'
                                            '        "io_kbytes" : 1639396,\n'
                                            '        "bw_bytes" : 239785959,\n'
                                            '        "bw" : 234165,\n'
                                            '        "iops" : 58541.494072,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 409849,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2966,\n'
                                            '          "max" : 24035,\n'
                                            '          "mean" : 3340.552930,\n'
                                            '          "stddev" : 206.631663,\n'
                                            '          "N" : 409849\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 1123,\n'
                                            '          "max" : 438051,\n'
                                            '          "mean" : 13173.901883,\n'
                                            '          "stddev" : '
                                            '2319.222617,\n'
                                            '          "N" : 409849,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 12096,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12224,\n'
                                            '            "30.000000" : 12352,\n'
                                            '            "40.000000" : 12480,\n'
                                            '            "50.000000" : 13376,\n'
                                            '            "60.000000" : 13376,\n'
                                            '            "70.000000" : 13376,\n'
                                            '            "80.000000" : 13504,\n'
                                            '            "90.000000" : 13504,\n'
                                            '            "95.000000" : 13632,\n'
                                            '            "99.000000" : 23680,\n'
                                            '            "99.500000" : 28800,\n'
                                            '            "99.900000" : 35072,\n'
                                            '            "99.950000" : 50944,\n'
                                            '            "99.990000" : 73216\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12203,\n'
                                            '          "max" : 441297,\n'
                                            '          "mean" : 16514.454814,\n'
                                            '          "stddev" : '
                                            '2343.548276,\n'
                                            '          "N" : 409849\n'
                                            '        },\n'
                                            '        "bw_min" : 226544,\n'
                                            '        "bw_max" : 245707,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '234225.357143,\n'
                                            '        "bw_dev" : 8659.960984,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 56636,\n'
                                            '        "iops_max" : 61426,\n'
                                            '        "iops_mean" : '
                                            '58556.285714,\n'
                                            '        "iops_stddev" : '
                                            '2164.913764,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 12.657143,\n'
                                            '      "sys_cpu" : 36.671429,\n'
                                            '      "ctx" : 409826,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.453089,\n'
                                            '        "50" : 1.489085,\n'
                                            '        "100" : 0.050506,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.010000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85718: Tue '
                                            'Oct 25 23:30:34 2022\n'
                                            '  read: IOPS=58.5k, BW=229MiB/s '
                                            '(240MB/s)(1601MiB/7001msec)\n'
                                            '    slat (nsec): min=2966, '
                                            'max=24035, avg=3340.55, '
                                            'stdev=206.63\n'
                                            '    clat (nsec): min=1123, '
                                            'max=438051, avg=13173.90, '
                                            'stdev=2319.22\n'
                                            '     lat (usec): min=12, max=441, '
                                            'avg=16.51, stdev= 2.34\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[12096], 10.00th=[12096], '
                                            '20.00th=[12224],\n'
                                            '     | 30.00th=[12352], '
                                            '40.00th=[12480], 50.00th=[13376], '
                                            '60.00th=[13376],\n'
                                            '     | 70.00th=[13376], '
                                            '80.00th=[13504], 90.00th=[13504], '
                                            '95.00th=[13632],\n'
                                            '     | 99.00th=[23680], '
                                            '99.50th=[28800], 99.90th=[35072], '
                                            '99.95th=[50944],\n'
                                            '     | 99.99th=[73216]\n'
                                            '   bw (  KiB/s): min=226544, '
                                            'max=245707, per=100.00%, '
                                            'avg=234225.36, stdev=8659.96, '
                                            'samples=14\n'
                                            '   iops        : min=56636, '
                                            'max=61426, avg=58556.29, '
                                            'stdev=2164.91, samples=14\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=0.01%, 20=98.45%, '
                                            '50=1.49%\n'
                                            '  lat (usec)   : 100=0.05%, '
                                            '250=0.01%, 500=0.01%\n'
                                            '  cpu          : usr=12.66%, '
                                            'sys=36.67%, ctx=409826, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=409849,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=229MiB/s (240MB/s), '
                                            '229MiB/s-229MiB/s '
                                            '(240MB/s-240MB/s), io=1601MiB '
                                            '(1679MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.output'),
                                  'state': {'begin': 1666726344.7567978,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_0.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.8069570064544678,
                                            'end': 1666726345.5637548,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_93.state')},
                       'cmd_94': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-xnvme_1.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.output'),
                                  'state': {'begin': 1666726346.6274297,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7909464836120605,
                                            'end': 1666726347.4183762,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_94.state')},
                       'cmd_95': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.output'),
                                  'state': {'begin': 1666726347.420631,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-xnvme_1.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.116731643676758,
                                            'end': 1666726358.5373626,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_95.state')},
                       'cmd_96': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 4096B-4096B, (W) '
                                            '4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733448,\n'
                                            '  "timestamp_ms" : '
                                            '1666733448033,\n'
                                            '  "time" : "Tue Oct 25 23:30:48 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1764671488,\n'
                                            '        "io_kbytes" : 1723312,\n'
                                            '        "bw_bytes" : 252059918,\n'
                                            '        "bw" : 246152,\n'
                                            '        "iops" : 61538.065991,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 430828,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2925,\n'
                                            '          "max" : 175278,\n'
                                            '          "mean" : 3159.793955,\n'
                                            '          "stddev" : 387.428894,\n'
                                            '          "N" : 430828\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 1002,\n'
                                            '          "max" : 182192,\n'
                                            '          "mean" : 12504.355956,\n'
                                            '          "stddev" : '
                                            '2223.696600,\n'
                                            '          "N" : 430828,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 11968,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12096,\n'
                                            '            "30.000000" : 12096,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12224,\n'
                                            '            "60.000000" : 12224,\n'
                                            '            "70.000000" : 12352,\n'
                                            '            "80.000000" : 12352,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12608,\n'
                                            '            "99.000000" : 22144,\n'
                                            '            "99.500000" : 29056,\n'
                                            '            "99.900000" : 37632,\n'
                                            '            "99.950000" : 52992,\n'
                                            '            "99.990000" : 73216\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12213,\n'
                                            '          "max" : 188343,\n'
                                            '          "mean" : 15664.149911,\n'
                                            '          "stddev" : '
                                            '2259.446273,\n'
                                            '          "N" : 430828\n'
                                            '        },\n'
                                            '        "bw_min" : 245008,\n'
                                            '        "bw_max" : 247408,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '246227.357143,\n'
                                            '        "bw_dev" : 707.156343,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 61252,\n'
                                            '        "iops_max" : 61852,\n'
                                            '        "iops_mean" : '
                                            '61556.785714,\n'
                                            '        "iops_stddev" : '
                                            '176.840727,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 13.242857,\n'
                                            '      "sys_cpu" : 39.485714,\n'
                                            '      "ctx" : 430779,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.010213,\n'
                                            '        "20" : 98.469227,\n'
                                            '        "50" : 1.455801,\n'
                                            '        "100" : 0.060581,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85758: Tue '
                                            'Oct 25 23:30:48 2022\n'
                                            '  read: IOPS=61.5k, BW=240MiB/s '
                                            '(252MB/s)(1683MiB/7001msec)\n'
                                            '    slat (nsec): min=2925, '
                                            'max=175278, avg=3159.79, '
                                            'stdev=387.43\n'
                                            '    clat (nsec): min=1002, '
                                            'max=182192, avg=12504.36, '
                                            'stdev=2223.70\n'
                                            '     lat (usec): min=12, max=188, '
                                            'avg=15.66, stdev= 2.26\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[11968], 10.00th=[12096], '
                                            '20.00th=[12096],\n'
                                            '     | 30.00th=[12096], '
                                            '40.00th=[12224], 50.00th=[12224], '
                                            '60.00th=[12224],\n'
                                            '     | 70.00th=[12352], '
                                            '80.00th=[12352], 90.00th=[12480], '
                                            '95.00th=[12608],\n'
                                            '     | 99.00th=[22144], '
                                            '99.50th=[29056], 99.90th=[37632], '
                                            '99.95th=[52992],\n'
                                            '     | 99.99th=[73216]\n'
                                            '   bw (  KiB/s): min=245008, '
                                            'max=247408, per=100.00%, '
                                            'avg=246227.36, stdev=707.16, '
                                            'samples=14\n'
                                            '   iops        : min=61252, '
                                            'max=61852, avg=61556.79, '
                                            'stdev=176.84, samples=14\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '10=0.01%, 20=98.47%, 50=1.46%, '
                                            '100=0.06%\n'
                                            '  lat (usec)   : 250=0.01%\n'
                                            '  cpu          : usr=13.24%, '
                                            'sys=39.49%, ctx=430779, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=430828,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=240MiB/s (252MB/s), '
                                            '240MiB/s-240MiB/s '
                                            '(252MB/s-252MB/s), io=1683MiB '
                                            '(1765MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.output'),
                                  'state': {'begin': 1666726358.5440695,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_1.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7939703464508057,
                                            'end': 1666726359.3380399,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_96.state')},
                       'cmd_97': {'output': 'rm: cannot remove '
                                            "'/tmp/fio-output_4K_1_libaio-xnvme_2.txt': "
                                            'No such file or directory\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.output'),
                                  'state': {'begin': 1666726360.324495,
                                            'cmd': 'rm '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.7884197235107422,
                                            'end': 1666726361.1129148,
                                            'err': 1,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_97.state')},
                       'cmd_98': {'output': '',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.output'),
                                  'state': {'begin': 1666726361.1157036,
                                            'cmd': '/opt/fio/bin/fio '
                                                   '--name="libaio-xnvme" '
                                                   '--rw="randread" '
                                                   '--size="12G" --iodepth="1" '
                                                   '--bs="4K" --direct="1" '
                                                   '--thread="1" '
                                                   '--time_based="1" '
                                                   '--runtime="7" '
                                                   '--ramp_time="3" '
                                                   '--norandommap="1" '
                                                   '--allow_file_create="0" '
                                                   '--ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" '
                                                   '--xnvme_be="linux" '
                                                   '--xnvme_async="libaio" '
                                                   '--xnvme_sync="nvme" '
                                                   '--xnvme_admin="nvme" '
                                                   '--xnvme_dev_nsid="1" '
                                                   '--filename="/dev/nvme0n1" '
                                                   '--output-format="normal,json" '
                                                   '--output="/tmp/fio-output_4K_1_libaio-xnvme_2.txt"',
                                            'cwd': 'None',
                                            'elapsed': 11.09414291381836,
                                            'end': 1666726372.2098465,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_98.state')},
                       'cmd_99': {'output': 'libaio-xnvme: (g=0): rw=randread, '
                                            'bs=(R) 4096B-4096B, (W) '
                                            '4096B-4096B, (T) 4096B-4096B, '
                                            'ioengine=xnvme, iodepth=1\n'
                                            'fio-3.32-dirty\n'
                                            'Starting 1 thread\n'
                                            '{\n'
                                            '  "fio version" : '
                                            '"fio-3.32-dirty",\n'
                                            '  "timestamp" : 1666733461,\n'
                                            '  "timestamp_ms" : '
                                            '1666733461706,\n'
                                            '  "time" : "Tue Oct 25 23:31:01 '
                                            '2022",\n'
                                            '  "jobs" : [\n'
                                            '    {\n'
                                            '      "jobname" : '
                                            '"libaio-xnvme",\n'
                                            '      "groupid" : 0,\n'
                                            '      "error" : 0,\n'
                                            '      "eta" : 0,\n'
                                            '      "elapsed" : 11,\n'
                                            '      "job options" : {\n'
                                            '        "name" : "libaio-xnvme",\n'
                                            '        "rw" : "randread",\n'
                                            '        "size" : "12G",\n'
                                            '        "iodepth" : "1",\n'
                                            '        "bs" : "4K",\n'
                                            '        "direct" : "1",\n'
                                            '        "thread" : "1",\n'
                                            '        "time_based" : "1",\n'
                                            '        "runtime" : "7",\n'
                                            '        "ramp_time" : "3",\n'
                                            '        "norandommap" : "1",\n'
                                            '        "allow_file_create" : '
                                            '"0",\n'
                                            '        "ioengine" : '
                                            '"external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",\n'
                                            '        "xnvme_be" : "linux",\n'
                                            '        "xnvme_async" : '
                                            '"libaio",\n'
                                            '        "xnvme_sync" : "nvme",\n'
                                            '        "xnvme_admin" : "nvme",\n'
                                            '        "xnvme_dev_nsid" : "1",\n'
                                            '        "filename" : '
                                            '"/dev/nvme0n1"\n'
                                            '      },\n'
                                            '      "read" : {\n'
                                            '        "io_bytes" : 1754910720,\n'
                                            '        "io_kbytes" : 1713780,\n'
                                            '        "bw_bytes" : 250665722,\n'
                                            '        "bw" : 244790,\n'
                                            '        "iops" : 61197.686045,\n'
                                            '        "runtime" : 7001,\n'
                                            '        "total_ios" : 428445,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 2905,\n'
                                            '          "max" : 25257,\n'
                                            '          "mean" : 3138.958107,\n'
                                            '          "stddev" : 264.649411,\n'
                                            '          "N" : 428445\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 1072,\n'
                                            '          "max" : 196789,\n'
                                            '          "mean" : 12600.985377,\n'
                                            '          "stddev" : '
                                            '2183.473992,\n'
                                            '          "N" : 428445,\n'
                                            '          "percentile" : {\n'
                                            '            "1.000000" : 11968,\n'
                                            '            "5.000000" : 12096,\n'
                                            '            "10.000000" : 12096,\n'
                                            '            "20.000000" : 12224,\n'
                                            '            "30.000000" : 12224,\n'
                                            '            "40.000000" : 12224,\n'
                                            '            "50.000000" : 12352,\n'
                                            '            "60.000000" : 12352,\n'
                                            '            "70.000000" : 12480,\n'
                                            '            "80.000000" : 12480,\n'
                                            '            "90.000000" : 12480,\n'
                                            '            "95.000000" : 12736,\n'
                                            '            "99.000000" : 22400,\n'
                                            '            "99.500000" : 29312,\n'
                                            '            "99.900000" : 32384,\n'
                                            '            "99.950000" : 50944,\n'
                                            '            "99.990000" : 72192\n'
                                            '          }\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 12002,\n'
                                            '          "max" : 199895,\n'
                                            '          "mean" : 15739.943484,\n'
                                            '          "stddev" : '
                                            '2203.074385,\n'
                                            '          "N" : 428445\n'
                                            '        },\n'
                                            '        "bw_min" : 243672,\n'
                                            '        "bw_max" : 246248,\n'
                                            '        "bw_agg" : 100.000000,\n'
                                            '        "bw_mean" : '
                                            '244847.571429,\n'
                                            '        "bw_dev" : 682.736655,\n'
                                            '        "bw_samples" : 14,\n'
                                            '        "iops_min" : 60918,\n'
                                            '        "iops_max" : 61562,\n'
                                            '        "iops_mean" : '
                                            '61211.857143,\n'
                                            '        "iops_stddev" : '
                                            '170.666502,\n'
                                            '        "iops_samples" : 14\n'
                                            '      },\n'
                                            '      "write" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "trim" : {\n'
                                            '        "io_bytes" : 0,\n'
                                            '        "io_kbytes" : 0,\n'
                                            '        "bw_bytes" : 0,\n'
                                            '        "bw" : 0,\n'
                                            '        "iops" : 0.000000,\n'
                                            '        "runtime" : 0,\n'
                                            '        "total_ios" : 0,\n'
                                            '        "short_ios" : 0,\n'
                                            '        "drop_ios" : 0,\n'
                                            '        "slat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "clat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        },\n'
                                            '        "bw_min" : 0,\n'
                                            '        "bw_max" : 0,\n'
                                            '        "bw_agg" : 0.000000,\n'
                                            '        "bw_mean" : 0.000000,\n'
                                            '        "bw_dev" : 0.000000,\n'
                                            '        "bw_samples" : 0,\n'
                                            '        "iops_min" : 0,\n'
                                            '        "iops_max" : 0,\n'
                                            '        "iops_mean" : 0.000000,\n'
                                            '        "iops_stddev" : '
                                            '0.000000,\n'
                                            '        "iops_samples" : 0\n'
                                            '      },\n'
                                            '      "sync" : {\n'
                                            '        "total_ios" : 0,\n'
                                            '        "lat_ns" : {\n'
                                            '          "min" : 0,\n'
                                            '          "max" : 0,\n'
                                            '          "mean" : 0.000000,\n'
                                            '          "stddev" : 0.000000,\n'
                                            '          "N" : 0\n'
                                            '        }\n'
                                            '      },\n'
                                            '      "job_runtime" : 7000,\n'
                                            '      "usr_cpu" : 12.700000,\n'
                                            '      "sys_cpu" : 39.757143,\n'
                                            '      "ctx" : 428394,\n'
                                            '      "majf" : 0,\n'
                                            '      "minf" : 0,\n'
                                            '      "iodepth_level" : {\n'
                                            '        "1" : 100.000000,\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_submit" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "iodepth_complete" : {\n'
                                            '        "0" : 0.000000,\n'
                                            '        "4" : 100.000000,\n'
                                            '        "8" : 0.000000,\n'
                                            '        "16" : 0.000000,\n'
                                            '        "32" : 0.000000,\n'
                                            '        "64" : 0.000000,\n'
                                            '        ">=64" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ns" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_us" : {\n'
                                            '        "2" : 0.010000,\n'
                                            '        "4" : 0.010000,\n'
                                            '        "10" : 0.010000,\n'
                                            '        "20" : 98.445308,\n'
                                            '        "50" : 1.491907,\n'
                                            '        "100" : 0.050415,\n'
                                            '        "250" : 0.010000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_ms" : {\n'
                                            '        "2" : 0.000000,\n'
                                            '        "4" : 0.000000,\n'
                                            '        "10" : 0.000000,\n'
                                            '        "20" : 0.000000,\n'
                                            '        "50" : 0.000000,\n'
                                            '        "100" : 0.000000,\n'
                                            '        "250" : 0.000000,\n'
                                            '        "500" : 0.000000,\n'
                                            '        "750" : 0.000000,\n'
                                            '        "1000" : 0.000000,\n'
                                            '        "2000" : 0.000000,\n'
                                            '        ">=2000" : 0.000000\n'
                                            '      },\n'
                                            '      "latency_depth" : 1,\n'
                                            '      "latency_target" : 0,\n'
                                            '      "latency_percentile" : '
                                            '100.000000,\n'
                                            '      "latency_window" : 0\n'
                                            '    }\n'
                                            '  ]\n'
                                            '}\n'
                                            '\n'
                                            'libaio-xnvme: (groupid=0, '
                                            'jobs=1): err= 0: pid=85798: Tue '
                                            'Oct 25 23:31:01 2022\n'
                                            '  read: IOPS=61.2k, BW=239MiB/s '
                                            '(251MB/s)(1674MiB/7001msec)\n'
                                            '    slat (nsec): min=2905, '
                                            'max=25257, avg=3138.96, '
                                            'stdev=264.65\n'
                                            '    clat (nsec): min=1072, '
                                            'max=196789, avg=12600.99, '
                                            'stdev=2183.47\n'
                                            '     lat (usec): min=12, max=199, '
                                            'avg=15.74, stdev= 2.20\n'
                                            '    clat percentiles (nsec):\n'
                                            '     |  1.00th=[11968],  '
                                            '5.00th=[12096], 10.00th=[12096], '
                                            '20.00th=[12224],\n'
                                            '     | 30.00th=[12224], '
                                            '40.00th=[12224], 50.00th=[12352], '
                                            '60.00th=[12352],\n'
                                            '     | 70.00th=[12480], '
                                            '80.00th=[12480], 90.00th=[12480], '
                                            '95.00th=[12736],\n'
                                            '     | 99.00th=[22400], '
                                            '99.50th=[29312], 99.90th=[32384], '
                                            '99.95th=[50944],\n'
                                            '     | 99.99th=[72192]\n'
                                            '   bw (  KiB/s): min=243672, '
                                            'max=246248, per=100.00%, '
                                            'avg=244847.57, stdev=682.74, '
                                            'samples=14\n'
                                            '   iops        : min=60918, '
                                            'max=61562, avg=61211.86, '
                                            'stdev=170.67, samples=14\n'
                                            '  lat (usec)   : 2=0.01%, '
                                            '4=0.01%, 10=0.01%, 20=98.45%, '
                                            '50=1.49%\n'
                                            '  lat (usec)   : 100=0.05%, '
                                            '250=0.01%\n'
                                            '  cpu          : usr=12.70%, '
                                            'sys=39.76%, ctx=428394, majf=0, '
                                            'minf=0\n'
                                            '  IO depths    : 1=100.0%, '
                                            '2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, >=64=0.0%\n'
                                            '     submit    : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     complete  : 0=0.0%, '
                                            '4=100.0%, 8=0.0%, 16=0.0%, '
                                            '32=0.0%, 64=0.0%, >=64=0.0%\n'
                                            '     issued rwts: '
                                            'total=428445,0,0,0 short=0,0,0,0 '
                                            'dropped=0,0,0,0\n'
                                            '     latency   : target=0, '
                                            'window=0, percentile=100.00%, '
                                            'depth=1\n'
                                            '\n'
                                            'Run status group 0 (all jobs):\n'
                                            '   READ: bw=239MiB/s (251MB/s), '
                                            '239MiB/s-239MiB/s '
                                            '(251MB/s-251MB/s), io=1674MiB '
                                            '(1755MB), run=7001-7001msec\n',
                                  'output_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.output'),
                                  'state': {'begin': 1666726372.2128544,
                                            'cmd': 'cat '
                                                   '/tmp/fio-output_4K_1_libaio-xnvme_2.txt',
                                            'cwd': 'None',
                                            'elapsed': 0.78910231590271,
                                            'end': 1666726373.0019567,
                                            'err': 0,
                                            'output_dpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio',
                                            'output_fpath': '/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.output'},
                                  'state_path': PosixPath('/home/safl/tdde/summit2022/cijoe-output/002_fio/cmd_99.state')}}},
 'id': '002_fio',
 'name': 'fio',
 'nr': 2,
 'status': {'elapsed': 1890.7196505069733,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'fioe',
 'with': {'iodepths': [1, 2, 4, 8], 'iosizes': ['4K'], 'repetitions': 3}}
$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733002,
  "timestamp_ms" : 1666733002638,
  "time" : "Tue Oct 25 23:23:22 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1880924160,
        "io_kbytes" : 1836840,
        "bw_bytes" : 268665070,
        "bw" : 262368,
        "iops" : 65591.915441,
        "runtime" : 7001,
        "total_ios" : 459209,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2184,
          "max" : 28192,
          "mean" : 2371.134109,
          "stddev" : 209.422890,
          "N" : 459210
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 1211963,
          "mean" : 12255.930946,
          "stddev" : 2652.769239,
          "N" : 459209,
          "percentile" : {
            "1.000000" : 11712,
            "5.000000" : 11712,
            "10.000000" : 11840,
            "20.000000" : 11840,
            "30.000000" : 11840,
            "40.000000" : 11968,
            "50.000000" : 11968,
            "60.000000" : 12096,
            "70.000000" : 12096,
            "80.000000" : 12096,
            "90.000000" : 12224,
            "95.000000" : 12352,
            "99.000000" : 21888,
            "99.500000" : 23424,
            "99.900000" : 31872,
            "99.950000" : 50432,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 11632,
          "max" : 1214328,
          "mean" : 14627.064896,
          "stddev" : 2660.860079,
          "N" : 459209
        },
        "bw_min" : 260360,
        "bw_max" : 263680,
        "bw_agg" : 100.000000,
        "bw_mean" : 262437.714286,
        "bw_dev" : 778.330702,
        "bw_samples" : 14,
        "iops_min" : 65090,
        "iops_max" : 65920,
        "iops_mean" : 65609.428571,
        "iops_stddev" : 194.582675,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.857143,
      "sys_cpu" : 36.157143,
      "ctx" : 459183,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.399857,
        "50" : 1.534595,
        "100" : 0.054877,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84392: Tue Oct 25 23:23:22 2022
  read: IOPS=65.6k, BW=256MiB/s (269MB/s)(1794MiB/7001msec)
    slat (nsec): min=2184, max=28192, avg=2371.13, stdev=209.42
    clat (nsec): min=241, max=1212.0k, avg=12255.93, stdev=2652.77
     lat (usec): min=11, max=1214, avg=14.63, stdev= 2.66
    clat percentiles (nsec):
     |  1.00th=[11712],  5.00th=[11712], 10.00th=[11840], 20.00th=[11840],
     | 30.00th=[11840], 40.00th=[11968], 50.00th=[11968], 60.00th=[12096],
     | 70.00th=[12096], 80.00th=[12096], 90.00th=[12224], 95.00th=[12352],
     | 99.00th=[21888], 99.50th=[23424], 99.90th=[31872], 99.95th=[50432],
     | 99.99th=[64256]
   bw (  KiB/s): min=260360, max=263680, per=100.00%, avg=262437.71, stdev=778.33, samples=14
   iops        : min=65090, max=65920, avg=65609.43, stdev=194.58, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.40%, 50=1.53%, 100=0.05%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=9.86%, sys=36.16%, ctx=459183, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=459209,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=256MiB/s (269MB/s), 256MiB/s-256MiB/s (269MB/s-269MB/s), io=1794MiB (1881MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733016,
  "timestamp_ms" : 1666733016245,
  "time" : "Tue Oct 25 23:23:36 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1887145984,
        "io_kbytes" : 1842916,
        "bw_bytes" : 269553775,
        "bw" : 263236,
        "iops" : 65809.027282,
        "runtime" : 7001,
        "total_ios" : 460729,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2174,
          "max" : 31970,
          "mean" : 2353.754324,
          "stddev" : 207.012099,
          "N" : 460729
        },
        "clat_ns" : {
          "min" : 200,
          "max" : 183274,
          "mean" : 12205.161577,
          "stddev" : 1978.440122,
          "N" : 460729,
          "percentile" : {
            "1.000000" : 11712,
            "5.000000" : 11712,
            "10.000000" : 11712,
            "20.000000" : 11840,
            "30.000000" : 11840,
            "40.000000" : 11840,
            "50.000000" : 11968,
            "60.000000" : 11968,
            "70.000000" : 12096,
            "80.000000" : 12096,
            "90.000000" : 12096,
            "95.000000" : 12352,
            "99.000000" : 21888,
            "99.500000" : 23168,
            "99.900000" : 31360,
            "99.950000" : 49920,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 11591,
          "max" : 185708,
          "mean" : 14558.915901,
          "stddev" : 1988.143930,
          "N" : 460729
        },
        "bw_min" : 262208,
        "bw_max" : 263920,
        "bw_agg" : 100.000000,
        "bw_mean" : 263299.928571,
        "bw_dev" : 429.320118,
        "bw_samples" : 14,
        "iops_min" : 65552,
        "iops_max" : 65980,
        "iops_mean" : 65824.928571,
        "iops_stddev" : 107.268509,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.728571,
      "sys_cpu" : 36.385714,
      "ctx" : 460697,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.392330,
        "50" : 1.542989,
        "100" : 0.051874,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84430: Tue Oct 25 23:23:36 2022
  read: IOPS=65.8k, BW=257MiB/s (270MB/s)(1800MiB/7001msec)
    slat (nsec): min=2174, max=31970, avg=2353.75, stdev=207.01
    clat (nsec): min=200, max=183274, avg=12205.16, stdev=1978.44
     lat (usec): min=11, max=185, avg=14.56, stdev= 1.99
    clat percentiles (nsec):
     |  1.00th=[11712],  5.00th=[11712], 10.00th=[11712], 20.00th=[11840],
     | 30.00th=[11840], 40.00th=[11840], 50.00th=[11968], 60.00th=[11968],
     | 70.00th=[12096], 80.00th=[12096], 90.00th=[12096], 95.00th=[12352],
     | 99.00th=[21888], 99.50th=[23168], 99.90th=[31360], 99.95th=[49920],
     | 99.99th=[64256]
   bw (  KiB/s): min=262208, max=263920, per=100.00%, avg=263299.93, stdev=429.32, samples=14
   iops        : min=65552, max=65980, avg=65824.93, stdev=107.27, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=0.01%, 20=98.39%, 50=1.54%, 100=0.05%, 250=0.01%
  cpu          : usr=9.73%, sys=36.39%, ctx=460697, majf=0, minf=1
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=460729,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=257MiB/s (270MB/s), 257MiB/s-257MiB/s (270MB/s-270MB/s), io=1800MiB (1887MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733029,
  "timestamp_ms" : 1666733029933,
  "time" : "Tue Oct 25 23:23:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1777991680,
        "io_kbytes" : 1736320,
        "bw_bytes" : 253962531,
        "bw" : 248010,
        "iops" : 62002.571061,
        "runtime" : 7001,
        "total_ios" : 434080,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2184,
          "max" : 31269,
          "mean" : 2476.576864,
          "stddev" : 207.606246,
          "N" : 434080
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 133119,
          "mean" : 13071.487523,
          "stddev" : 2165.425154,
          "N" : 434080,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 11968,
            "10.000000" : 12096,
            "20.000000" : 12096,
            "30.000000" : 12096,
            "40.000000" : 12224,
            "50.000000" : 13248,
            "60.000000" : 13376,
            "70.000000" : 13376,
            "80.000000" : 13376,
            "90.000000" : 13504,
            "95.000000" : 13632,
            "99.000000" : 22912,
            "99.500000" : 29312,
            "99.900000" : 32640,
            "99.950000" : 50432,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 13816,
          "max" : 135424,
          "mean" : 15548.064387,
          "stddev" : 2206.175957,
          "N" : 434080
        },
        "bw_min" : 237664,
        "bw_max" : 261000,
        "bw_agg" : 100.000000,
        "bw_mean" : 248095.357143,
        "bw_dev" : 11422.448246,
        "bw_samples" : 14,
        "iops_min" : 59416,
        "iops_max" : 65250,
        "iops_mean" : 62023.785714,
        "iops_stddev" : 2855.663204,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.014286,
      "sys_cpu" : 35.057143,
      "ctx" : 434068,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.284187,
        "50" : 1.652691,
        "100" : 0.052525,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=84468: Tue Oct 25 23:23:49 2022
  read: IOPS=62.0k, BW=242MiB/s (254MB/s)(1696MiB/7001msec)
    slat (nsec): min=2184, max=31269, avg=2476.58, stdev=207.61
    clat (nsec): min=250, max=133119, avg=13071.49, stdev=2165.43
     lat (usec): min=13, max=135, avg=15.55, stdev= 2.21
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[11968], 10.00th=[12096], 20.00th=[12096],
     | 30.00th=[12096], 40.00th=[12224], 50.00th=[13248], 60.00th=[13376],
     | 70.00th=[13376], 80.00th=[13376], 90.00th=[13504], 95.00th=[13632],
     | 99.00th=[22912], 99.50th=[29312], 99.90th=[32640], 99.95th=[50432],
     | 99.99th=[68096]
   bw (  KiB/s): min=237664, max=261000, per=100.00%, avg=248095.36, stdev=11422.45, samples=14
   iops        : min=59416, max=65250, avg=62023.79, stdev=2855.66, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.01%, 20=98.28%, 50=1.65%, 100=0.05%, 250=0.01%
  cpu          : usr=8.01%, sys=35.06%, ctx=434068, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=434080,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=242MiB/s (254MB/s), 242MiB/s-242MiB/s (254MB/s-254MB/s), io=1696MiB (1778MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt': No such file or directory

$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733475,
  "timestamp_ms" : 1666733475316,
  "time" : "Tue Oct 25 23:31:15 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2836426752,
        "io_kbytes" : 2769948,
        "bw_bytes" : 405145943,
        "bw" : 395650,
        "iops" : 98912.298243,
        "runtime" : 7001,
        "total_ios" : 692485,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1893,
          "max" : 27962,
          "mean" : 2200.003763,
          "stddev" : 323.295581,
          "N" : 692486
        },
        "clat_ns" : {
          "min" : 3106,
          "max" : 189856,
          "mean" : 17485.482183,
          "stddev" : 3150.118639,
          "N" : 692486,
          "percentile" : {
            "1.000000" : 16320,
            "5.000000" : 16512,
            "10.000000" : 16512,
            "20.000000" : 16512,
            "30.000000" : 16512,
            "40.000000" : 16768,
            "50.000000" : 17024,
            "60.000000" : 17024,
            "70.000000" : 17024,
            "80.000000" : 17280,
            "90.000000" : 17280,
            "95.000000" : 19584,
            "99.000000" : 33536,
            "99.500000" : 34560,
            "99.900000" : 50432,
            "99.950000" : 62720,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 11261,
          "max" : 191810,
          "mean" : 19685.485946,
          "stddev" : 3142.083143,
          "N" : 692486
        },
        "bw_min" : 394512,
        "bw_max" : 396576,
        "bw_agg" : 100.000000,
        "bw_mean" : 395746.285714,
        "bw_dev" : 618.185363,
        "bw_samples" : 14,
        "iops_min" : 98628,
        "iops_max" : 99144,
        "iops_mean" : 98936.571429,
        "iops_stddev" : 154.546341,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.085714,
      "sys_cpu" : 38.200000,
      "ctx" : 346121,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010686,
        "20" : 95.026318,
        "50" : 4.855268,
        "100" : 0.104840,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85837: Tue Oct 25 23:31:15 2022
  read: IOPS=98.9k, BW=386MiB/s (405MB/s)(2705MiB/7001msec)
    slat (nsec): min=1893, max=27962, avg=2200.00, stdev=323.30
    clat (usec): min=3, max=189, avg=17.49, stdev= 3.15
     lat (usec): min=11, max=191, avg=19.69, stdev= 3.14
    clat percentiles (nsec):
     |  1.00th=[16320],  5.00th=[16512], 10.00th=[16512], 20.00th=[16512],
     | 30.00th=[16512], 40.00th=[16768], 50.00th=[17024], 60.00th=[17024],
     | 70.00th=[17024], 80.00th=[17280], 90.00th=[17280], 95.00th=[19584],
     | 99.00th=[33536], 99.50th=[34560], 99.90th=[50432], 99.95th=[62720],
     | 99.99th=[73216]
   bw (  KiB/s): min=394512, max=396576, per=100.00%, avg=395746.29, stdev=618.19, samples=14
   iops        : min=98628, max=99144, avg=98936.57, stdev=154.55, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=95.03%, 50=4.86%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=12.09%, sys=38.20%, ctx=346121, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=692485,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=386MiB/s (405MB/s), 386MiB/s-386MiB/s (405MB/s-405MB/s), io=2705MiB (2836MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733488,
  "timestamp_ms" : 1666733488966,
  "time" : "Tue Oct 25 23:31:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2790170624,
        "io_kbytes" : 2724776,
        "bw_bytes" : 398538869,
        "bw" : 389198,
        "iops" : 97299.242965,
        "runtime" : 7001,
        "total_ios" : 681192,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1963,
          "max" : 37029,
          "mean" : 2279.325263,
          "stddev" : 339.765872,
          "N" : 681193
        },
        "clat_ns" : {
          "min" : 3075,
          "max" : 121388,
          "mean" : 17720.137531,
          "stddev" : 3211.933311,
          "N" : 681193,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16512,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 17280,
            "60.000000" : 17280,
            "70.000000" : 17280,
            "80.000000" : 17536,
            "90.000000" : 17536,
            "95.000000" : 20096,
            "99.000000" : 34048,
            "99.500000" : 35072,
            "99.900000" : 50432,
            "99.950000" : 63232,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11381,
          "max" : 123462,
          "mean" : 19999.462794,
          "stddev" : 3203.796715,
          "N" : 681193
        },
        "bw_min" : 388424,
        "bw_max" : 390448,
        "bw_agg" : 100.000000,
        "bw_mean" : 389292.714286,
        "bw_dev" : 718.143593,
        "bw_samples" : 14,
        "iops_min" : 97106,
        "iops_max" : 97612,
        "iops_mean" : 97323.142857,
        "iops_stddev" : 179.542520,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.500000,
      "sys_cpu" : 39.285714,
      "ctx" : 340448,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.011304,
        "20" : 94.985261,
        "50" : 4.893921,
        "100" : 0.104082,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85875: Tue Oct 25 23:31:28 2022
  read: IOPS=97.3k, BW=380MiB/s (399MB/s)(2661MiB/7001msec)
    slat (nsec): min=1963, max=37029, avg=2279.33, stdev=339.77
    clat (usec): min=3, max=121, avg=17.72, stdev= 3.21
     lat (usec): min=11, max=123, avg=20.00, stdev= 3.20
    clat percentiles (nsec):
     |  1.00th=[16512],  5.00th=[16512], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[17280], 60.00th=[17280],
     | 70.00th=[17280], 80.00th=[17536], 90.00th=[17536], 95.00th=[20096],
     | 99.00th=[34048], 99.50th=[35072], 99.90th=[50432], 99.95th=[63232],
     | 99.99th=[74240]
   bw (  KiB/s): min=388424, max=390448, per=100.00%, avg=389292.71, stdev=718.14, samples=14
   iops        : min=97106, max=97612, avg=97323.14, stdev=179.54, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=94.99%, 50=4.89%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=11.50%, sys=39.29%, ctx=340448, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=681192,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=380MiB/s (399MB/s), 380MiB/s-380MiB/s (399MB/s-399MB/s), io=2661MiB (2790MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733502,
  "timestamp_ms" : 1666733502622,
  "time" : "Tue Oct 25 23:31:42 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2812887040,
        "io_kbytes" : 2746960,
        "bw_bytes" : 401783608,
        "bw" : 392366,
        "iops" : 98091.558349,
        "runtime" : 7001,
        "total_ios" : 686739,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1884,
          "max" : 31248,
          "mean" : 2222.947482,
          "stddev" : 326.104416,
          "N" : 686739
        },
        "clat_ns" : {
          "min" : 2996,
          "max" : 255970,
          "mean" : 17635.028845,
          "stddev" : 3256.648995,
          "N" : 686740,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16512,
            "10.000000" : 16512,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 17024,
            "60.000000" : 17280,
            "70.000000" : 17280,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 20096,
            "99.000000" : 34048,
            "99.500000" : 34560,
            "99.900000" : 50944,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11231,
          "max" : 257904,
          "mean" : 19857.976051,
          "stddev" : 3249.803901,
          "N" : 686740
        },
        "bw_min" : 388705,
        "bw_max" : 395896,
        "bw_agg" : 100.000000,
        "bw_mean" : 392460.642857,
        "bw_dev" : 2587.148404,
        "bw_samples" : 14,
        "iops_min" : 97176,
        "iops_max" : 98974,
        "iops_mean" : 98115.142857,
        "iops_stddev" : 646.815020,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.271429,
      "sys_cpu" : 38.271429,
      "ctx" : 343225,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.014416,
        "20" : 94.981354,
        "50" : 4.886718,
        "100" : 0.113435,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=85913: Tue Oct 25 23:31:42 2022
  read: IOPS=98.1k, BW=383MiB/s (402MB/s)(2683MiB/7001msec)
    slat (nsec): min=1884, max=31248, avg=2222.95, stdev=326.10
    clat (usec): min=2, max=255, avg=17.64, stdev= 3.26
     lat (usec): min=11, max=257, avg=19.86, stdev= 3.25
    clat percentiles (nsec):
     |  1.00th=[16512],  5.00th=[16512], 10.00th=[16512], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[17024], 60.00th=[17280],
     | 70.00th=[17280], 80.00th=[17280], 90.00th=[17536], 95.00th=[20096],
     | 99.00th=[34048], 99.50th=[34560], 99.90th=[50944], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=388705, max=395896, per=100.00%, avg=392460.64, stdev=2587.15, samples=14
   iops        : min=97176, max=98974, avg=98115.14, stdev=646.82, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=94.98%, 50=4.89%, 100=0.11%
  lat (usec)   : 250=0.01%, 500=0.01%
  cpu          : usr=12.27%, sys=38.27%, ctx=343225, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=686739,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=383MiB/s (402MB/s), 383MiB/s-383MiB/s (402MB/s-402MB/s), io=2683MiB (2813MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt"
$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733516,
  "timestamp_ms" : 1666733516277,
  "time" : "Tue Oct 25 23:31:56 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2816339968,
        "io_kbytes" : 2750332,
        "bw_bytes" : 402276813,
        "bw" : 392848,
        "iops" : 98211.969719,
        "runtime" : 7001,
        "total_ios" : 687582,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2003,
          "max" : 40265,
          "mean" : 2351.189732,
          "stddev" : 337.786924,
          "N" : 687582
        },
        "clat_ns" : {
          "min" : 3356,
          "max" : 209222,
          "mean" : 17502.922175,
          "stddev" : 3232.779953,
          "N" : 687583,
          "percentile" : {
            "1.000000" : 16320,
            "5.000000" : 16512,
            "10.000000" : 16512,
            "20.000000" : 16512,
            "30.000000" : 16512,
            "40.000000" : 16768,
            "50.000000" : 17024,
            "60.000000" : 17024,
            "70.000000" : 17280,
            "80.000000" : 17280,
            "90.000000" : 17280,
            "95.000000" : 19584,
            "99.000000" : 33536,
            "99.500000" : 34560,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11251,
          "max" : 211306,
          "mean" : 19854.111460,
          "stddev" : 3223.695868,
          "N" : 687583
        },
        "bw_min" : 391559,
        "bw_max" : 394072,
        "bw_agg" : 100.000000,
        "bw_mean" : 392940.500000,
        "bw_dev" : 744.587315,
        "bw_samples" : 14,
        "iops_min" : 97889,
        "iops_max" : 98520,
        "iops_mean" : 98235.214286,
        "iops_stddev" : 186.488267,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.885714,
      "sys_cpu" : 39.400000,
      "ctx" : 343653,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010471,
        "20" : 95.019649,
        "50" : 4.858766,
        "100" : 0.108642,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=85953: Tue Oct 25 23:31:56 2022
  read: IOPS=98.2k, BW=384MiB/s (402MB/s)(2686MiB/7001msec)
    slat (nsec): min=2003, max=40265, avg=2351.19, stdev=337.79
    clat (usec): min=3, max=209, avg=17.50, stdev= 3.23
     lat (usec): min=11, max=211, avg=19.85, stdev= 3.22
    clat percentiles (nsec):
     |  1.00th=[16320],  5.00th=[16512], 10.00th=[16512], 20.00th=[16512],
     | 30.00th=[16512], 40.00th=[16768], 50.00th=[17024], 60.00th=[17024],
     | 70.00th=[17280], 80.00th=[17280], 90.00th=[17280], 95.00th=[19584],
     | 99.00th=[33536], 99.50th=[34560], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=391559, max=394072, per=100.00%, avg=392940.50, stdev=744.59, samples=14
   iops        : min=97889, max=98520, avg=98235.21, stdev=186.49, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=95.02%, 50=4.86%, 100=0.11%
  lat (usec)   : 250=0.01%
  cpu          : usr=12.89%, sys=39.40%, ctx=343653, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=687582,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=384MiB/s (402MB/s), 384MiB/s-384MiB/s (402MB/s-402MB/s), io=2686MiB (2816MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733529,
  "timestamp_ms" : 1666733529910,
  "time" : "Tue Oct 25 23:32:09 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2630627328,
        "io_kbytes" : 2568972,
        "bw_bytes" : 375750225,
        "bw" : 366943,
        "iops" : 91735.752035,
        "runtime" : 7001,
        "total_ios" : 642242,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2124,
          "max" : 14818,
          "mean" : 2496.051688,
          "stddev" : 247.656662,
          "N" : 642242
        },
        "clat_ns" : {
          "min" : 4859,
          "max" : 136816,
          "mean" : 18815.614652,
          "stddev" : 3328.463438,
          "N" : 642243,
          "percentile" : {
            "1.000000" : 17536,
            "5.000000" : 17792,
            "10.000000" : 17792,
            "20.000000" : 17792,
            "30.000000" : 17792,
            "40.000000" : 18048,
            "50.000000" : 18304,
            "60.000000" : 18304,
            "70.000000" : 18304,
            "80.000000" : 18560,
            "90.000000" : 18560,
            "95.000000" : 21888,
            "99.000000" : 35072,
            "99.500000" : 44288,
            "99.900000" : 49920,
            "99.950000" : 62208,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 12043,
          "max" : 139101,
          "mean" : 21311.665947,
          "stddev" : 3318.507739,
          "N" : 642243
        },
        "bw_min" : 366640,
        "bw_max" : 368008,
        "bw_agg" : 100.000000,
        "bw_mean" : 367114.357143,
        "bw_dev" : 409.196136,
        "bw_samples" : 14,
        "iops_min" : 91660,
        "iops_max" : 92002,
        "iops_mean" : 91778.500000,
        "iops_stddev" : 102.304034,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.457143,
      "sys_cpu" : 36.600000,
      "ctx" : 321062,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 94.633643,
        "50" : 5.261257,
        "100" : 0.097627,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=85992: Tue Oct 25 23:32:09 2022
  read: IOPS=91.7k, BW=358MiB/s (376MB/s)(2509MiB/7001msec)
    slat (nsec): min=2124, max=14818, avg=2496.05, stdev=247.66
    clat (usec): min=4, max=136, avg=18.82, stdev= 3.33
     lat (usec): min=12, max=139, avg=21.31, stdev= 3.32
    clat percentiles (nsec):
     |  1.00th=[17536],  5.00th=[17792], 10.00th=[17792], 20.00th=[17792],
     | 30.00th=[17792], 40.00th=[18048], 50.00th=[18304], 60.00th=[18304],
     | 70.00th=[18304], 80.00th=[18560], 90.00th=[18560], 95.00th=[21888],
     | 99.00th=[35072], 99.50th=[44288], 99.90th=[49920], 99.95th=[62208],
     | 99.99th=[74240]
   bw (  KiB/s): min=366640, max=368008, per=100.00%, avg=367114.36, stdev=409.20, samples=14
   iops        : min=91660, max=92002, avg=91778.50, stdev=102.30, samples=14
  lat (usec)   : 10=0.01%, 20=94.63%, 50=5.26%, 100=0.10%, 250=0.01%
  cpu          : usr=12.46%, sys=36.60%, ctx=321062, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=642242,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=358MiB/s (376MB/s), 358MiB/s-358MiB/s (376MB/s-376MB/s), io=2509MiB (2631MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733543,
  "timestamp_ms" : 1666733543542,
  "time" : "Tue Oct 25 23:32:23 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 2749571072,
        "io_kbytes" : 2685128,
        "bw_bytes" : 392739761,
        "bw" : 383534,
        "iops" : 95883.588059,
        "runtime" : 7001,
        "total_ios" : 671281,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2064,
          "max" : 31810,
          "mean" : 2400.967224,
          "stddev" : 306.105301,
          "N" : 671281
        },
        "clat_ns" : {
          "min" : 391,
          "max" : 135253,
          "mean" : 17952.067264,
          "stddev" : 3287.475107,
          "N" : 671282,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16512,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 17280,
            "50.000000" : 17280,
            "60.000000" : 17280,
            "70.000000" : 17792,
            "80.000000" : 18048,
            "90.000000" : 18304,
            "95.000000" : 20352,
            "99.000000" : 34048,
            "99.500000" : 37120,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11592,
          "max" : 143409,
          "mean" : 20353.034373,
          "stddev" : 3289.918264,
          "N" : 671282
        },
        "bw_min" : 367224,
        "bw_max" : 390992,
        "bw_agg" : 100.000000,
        "bw_mean" : 383631.928571,
        "bw_dev" : 9319.619474,
        "bw_samples" : 14,
        "iops_min" : 91806,
        "iops_max" : 97748,
        "iops_mean" : 95907.928571,
        "iops_stddev" : 2330.003977,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.528571,
      "sys_cpu" : 37.685714,
      "ctx" : 335517,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.011471,
        "20" : 94.923586,
        "50" : 4.958132,
        "100" : 0.104278,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=86031: Tue Oct 25 23:32:23 2022
  read: IOPS=95.9k, BW=375MiB/s (393MB/s)(2622MiB/7001msec)
    slat (nsec): min=2064, max=31810, avg=2400.97, stdev=306.11
    clat (nsec): min=391, max=135253, avg=17952.07, stdev=3287.48
     lat (usec): min=11, max=143, avg=20.35, stdev= 3.29
    clat percentiles (nsec):
     |  1.00th=[16512],  5.00th=[16512], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[17280], 50.00th=[17280], 60.00th=[17280],
     | 70.00th=[17792], 80.00th=[18048], 90.00th=[18304], 95.00th=[20352],
     | 99.00th=[34048], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=367224, max=390992, per=100.00%, avg=383631.93, stdev=9319.62, samples=14
   iops        : min=91806, max=97748, avg=95907.93, stdev=2330.00, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=94.92%, 50=4.96%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.53%, sys=37.69%, ctx=335517, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=671281,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=375MiB/s (393MB/s), 375MiB/s-375MiB/s (393MB/s-393MB/s), io=2622MiB (2750MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733043,
  "timestamp_ms" : 1666733043592,
  "time" : "Tue Oct 25 23:24:03 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1862557696,
        "io_kbytes" : 1818904,
        "bw_bytes" : 266041664,
        "bw" : 259806,
        "iops" : 64951.578346,
        "runtime" : 7001,
        "total_ios" : 454726,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2314,
          "max" : 40516,
          "mean" : 2487.807033,
          "stddev" : 208.501404,
          "N" : 454726
        },
        "clat_ns" : {
          "min" : 351,
          "max" : 1341115,
          "mean" : 12323.750791,
          "stddev" : 2811.725511,
          "N" : 454726,
          "percentile" : {
            "1.000000" : 11840,
            "5.000000" : 11840,
            "10.000000" : 11840,
            "20.000000" : 11840,
            "30.000000" : 11968,
            "40.000000" : 11968,
            "50.000000" : 12096,
            "60.000000" : 12096,
            "70.000000" : 12096,
            "80.000000" : 12224,
            "90.000000" : 12224,
            "95.000000" : 12480,
            "99.000000" : 22144,
            "99.500000" : 23680,
            "99.900000" : 31616,
            "99.950000" : 50432,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 12694,
          "max" : 1343630,
          "mean" : 14811.557824,
          "stddev" : 2820.004980,
          "N" : 454726
        },
        "bw_min" : 259104,
        "bw_max" : 260440,
        "bw_agg" : 100.000000,
        "bw_mean" : 259877.071429,
        "bw_dev" : 357.766676,
        "bw_samples" : 14,
        "iops_min" : 64776,
        "iops_max" : 65110,
        "iops_mean" : 64969.214286,
        "iops_stddev" : 89.484144,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.557143,
      "sys_cpu" : 35.457143,
      "ctx" : 454702,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.417728,
        "50" : 1.522235,
        "100" : 0.051679,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84507: Tue Oct 25 23:24:03 2022
  read: IOPS=65.0k, BW=254MiB/s (266MB/s)(1776MiB/7001msec)
    slat (nsec): min=2314, max=40516, avg=2487.81, stdev=208.50
    clat (nsec): min=351, max=1341.1k, avg=12323.75, stdev=2811.73
     lat (usec): min=12, max=1343, avg=14.81, stdev= 2.82
    clat percentiles (nsec):
     |  1.00th=[11840],  5.00th=[11840], 10.00th=[11840], 20.00th=[11840],
     | 30.00th=[11968], 40.00th=[11968], 50.00th=[12096], 60.00th=[12096],
     | 70.00th=[12096], 80.00th=[12224], 90.00th=[12224], 95.00th=[12480],
     | 99.00th=[22144], 99.50th=[23680], 99.90th=[31616], 99.95th=[50432],
     | 99.99th=[64256]
   bw (  KiB/s): min=259104, max=260440, per=100.00%, avg=259877.07, stdev=357.77, samples=14
   iops        : min=64776, max=65110, avg=64969.21, stdev=89.48, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 10=0.01%, 20=98.42%, 50=1.52%, 100=0.05%, 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=11.56%, sys=35.46%, ctx=454702, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=454726,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=1776MiB (1863MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733558,
  "timestamp_ms" : 1666733558546,
  "time" : "Tue Oct 25 23:32:38 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3217563648,
        "io_kbytes" : 3142152,
        "bw_bytes" : 459586294,
        "bw" : 448814,
        "iops" : 112203.542351,
        "runtime" : 7001,
        "total_ios" : 785537,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 661,
          "max" : 7804,
          "mean" : 916.427430,
          "stddev" : 81.567851,
          "N" : 785537
        },
        "clat_ns" : {
          "min" : 7284,
          "max" : 134732,
          "mean" : 16623.412960,
          "stddev" : 3660.577461,
          "N" : 785538,
          "percentile" : {
            "1.000000" : 13632,
            "5.000000" : 13888,
            "10.000000" : 14016,
            "20.000000" : 14144,
            "30.000000" : 14272,
            "40.000000" : 14400,
            "50.000000" : 17280,
            "60.000000" : 17536,
            "70.000000" : 17792,
            "80.000000" : 18048,
            "90.000000" : 18048,
            "95.000000" : 19584,
            "99.000000" : 30848,
            "99.500000" : 34560,
            "99.900000" : 51456,
            "99.950000" : 65280,
            "99.990000" : 79360
          }
        },
        "lat_ns" : {
          "min" : 11752,
          "max" : 135514,
          "mean" : 17539.840385,
          "stddev" : 3659.715140,
          "N" : 785538
        },
        "bw_min" : 447328,
        "bw_max" : 451014,
        "bw_agg" : 100.000000,
        "bw_mean" : 448941.000000,
        "bw_dev" : 1118.741257,
        "bw_samples" : 14,
        "iops_min" : 111832,
        "iops_max" : 112753,
        "iops_mean" : 112235.357143,
        "iops_stddev" : 279.439991,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 2,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 99.983196,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 95.054466,
        "50" : 4.836819,
        "100" : 0.108334,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86080: Tue Oct 25 23:32:38 2022
  read: IOPS=112k, BW=438MiB/s (460MB/s)(3069MiB/7001msec)
    slat (nsec): min=661, max=7804, avg=916.43, stdev=81.57
    clat (usec): min=7, max=134, avg=16.62, stdev= 3.66
     lat (usec): min=11, max=135, avg=17.54, stdev= 3.66
    clat percentiles (nsec):
     |  1.00th=[13632],  5.00th=[13888], 10.00th=[14016], 20.00th=[14144],
     | 30.00th=[14272], 40.00th=[14400], 50.00th=[17280], 60.00th=[17536],
     | 70.00th=[17792], 80.00th=[18048], 90.00th=[18048], 95.00th=[19584],
     | 99.00th=[30848], 99.50th=[34560], 99.90th=[51456], 99.95th=[65280],
     | 99.99th=[79360]
   bw (  KiB/s): min=447328, max=451014, per=100.00%, avg=448941.00, stdev=1118.74, samples=14
   iops        : min=111832, max=112753, avg=112235.36, stdev=279.44, samples=14
  lat (usec)   : 10=0.01%, 20=95.05%, 50=4.84%, 100=0.11%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=2, majf=0, minf=0
  IO depths    : 1=0.1%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=785537,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=438MiB/s (460MB/s), 438MiB/s-438MiB/s (460MB/s-460MB/s), io=3069MiB (3218MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733573,
  "timestamp_ms" : 1666733573691,
  "time" : "Tue Oct 25 23:32:53 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3401510912,
        "io_kbytes" : 3321788,
        "bw_bytes" : 485860721,
        "bw" : 474473,
        "iops" : 118618.197400,
        "runtime" : 7001,
        "total_ios" : 830446,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 671,
          "max" : 11402,
          "mean" : 967.528109,
          "stddev" : 130.573892,
          "N" : 830446
        },
        "clat_ns" : {
          "min" : 7574,
          "max" : 176641,
          "mean" : 15406.331199,
          "stddev" : 3395.306415,
          "N" : 830447,
          "percentile" : {
            "1.000000" : 13632,
            "5.000000" : 13760,
            "10.000000" : 13888,
            "20.000000" : 14656,
            "30.000000" : 14784,
            "40.000000" : 14784,
            "50.000000" : 14912,
            "60.000000" : 15040,
            "70.000000" : 15040,
            "80.000000" : 15168,
            "90.000000" : 15296,
            "95.000000" : 15808,
            "99.000000" : 31104,
            "99.500000" : 36096,
            "99.900000" : 51456,
            "99.950000" : 65280,
            "99.990000" : 82432
          }
        },
        "lat_ns" : {
          "min" : 10650,
          "max" : 177563,
          "mean" : 16373.859542,
          "stddev" : 3397.897104,
          "N" : 830447
        },
        "bw_min" : 473232,
        "bw_max" : 475744,
        "bw_agg" : 100.000000,
        "bw_mean" : 474607.785714,
        "bw_dev" : 803.230801,
        "bw_samples" : 14,
        "iops_min" : 118308,
        "iops_max" : 118936,
        "iops_mean" : 118651.928571,
        "iops_stddev" : 200.807778,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 15.182565,
        "2" : 84.817435,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 95.736147,
        "50" : 4.161378,
        "100" : 0.099224,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86130: Tue Oct 25 23:32:53 2022
  read: IOPS=119k, BW=463MiB/s (486MB/s)(3244MiB/7001msec)
    slat (nsec): min=671, max=11402, avg=967.53, stdev=130.57
    clat (usec): min=7, max=176, avg=15.41, stdev= 3.40
     lat (usec): min=10, max=177, avg=16.37, stdev= 3.40
    clat percentiles (nsec):
     |  1.00th=[13632],  5.00th=[13760], 10.00th=[13888], 20.00th=[14656],
     | 30.00th=[14784], 40.00th=[14784], 50.00th=[14912], 60.00th=[15040],
     | 70.00th=[15040], 80.00th=[15168], 90.00th=[15296], 95.00th=[15808],
     | 99.00th=[31104], 99.50th=[36096], 99.90th=[51456], 99.95th=[65280],
     | 99.99th=[82432]
   bw (  KiB/s): min=473232, max=475744, per=100.00%, avg=474607.79, stdev=803.23, samples=14
   iops        : min=118308, max=118936, avg=118651.93, stdev=200.81, samples=14
  lat (usec)   : 10=0.01%, 20=95.74%, 50=4.16%, 100=0.10%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=7, majf=0, minf=0
  IO depths    : 1=15.2%, 2=84.8%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=830446,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=463MiB/s (486MB/s), 463MiB/s-463MiB/s (486MB/s-486MB/s), io=3244MiB (3402MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733588,
  "timestamp_ms" : 1666733588818,
  "time" : "Tue Oct 25 23:33:08 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3410411520,
        "io_kbytes" : 3330480,
        "bw_bytes" : 487132055,
        "bw" : 475714,
        "iops" : 118928.581631,
        "runtime" : 7001,
        "total_ios" : 832619,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 661,
          "max" : 50916,
          "mean" : 946.325844,
          "stddev" : 169.673091,
          "N" : 832619
        },
        "clat_ns" : {
          "min" : 881,
          "max" : 1304386,
          "mean" : 15275.264800,
          "stddev" : 3935.692281,
          "N" : 832620,
          "percentile" : {
            "1.000000" : 13504,
            "5.000000" : 13632,
            "10.000000" : 13760,
            "20.000000" : 13888,
            "30.000000" : 14656,
            "40.000000" : 14784,
            "50.000000" : 14784,
            "60.000000" : 14912,
            "70.000000" : 15040,
            "80.000000" : 15040,
            "90.000000" : 15296,
            "95.000000" : 15808,
            "99.000000" : 31104,
            "99.500000" : 36608,
            "99.900000" : 48896,
            "99.950000" : 64768,
            "99.990000" : 82432
          }
        },
        "lat_ns" : {
          "min" : 10650,
          "max" : 1305568,
          "mean" : 16221.590603,
          "stddev" : 3939.658232,
          "N" : 832620
        },
        "bw_min" : 474520,
        "bw_max" : 477072,
        "bw_agg" : 100.000000,
        "bw_mean" : 475836.000000,
        "bw_dev" : 739.461759,
        "bw_samples" : 14,
        "iops_min" : 118630,
        "iops_max" : 119268,
        "iops_mean" : 118959.000000,
        "iops_stddev" : 184.865440,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 19,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 25.146916,
        "2" : 74.853084,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 95.767812,
        "50" : 4.132622,
        "100" : 0.094521,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86180: Tue Oct 25 23:33:08 2022
  read: IOPS=119k, BW=465MiB/s (487MB/s)(3252MiB/7001msec)
    slat (nsec): min=661, max=50916, avg=946.33, stdev=169.67
    clat (nsec): min=881, max=1304.4k, avg=15275.26, stdev=3935.69
     lat (usec): min=10, max=1305, avg=16.22, stdev= 3.94
    clat percentiles (nsec):
     |  1.00th=[13504],  5.00th=[13632], 10.00th=[13760], 20.00th=[13888],
     | 30.00th=[14656], 40.00th=[14784], 50.00th=[14784], 60.00th=[14912],
     | 70.00th=[15040], 80.00th=[15040], 90.00th=[15296], 95.00th=[15808],
     | 99.00th=[31104], 99.50th=[36608], 99.90th=[48896], 99.95th=[64768],
     | 99.99th=[82432]
   bw (  KiB/s): min=474520, max=477072, per=100.00%, avg=475836.00, stdev=739.46, samples=14
   iops        : min=118630, max=119268, avg=118959.00, stdev=184.87, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=95.77%, 50=4.13%
  lat (usec)   : 100=0.09%, 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=19, majf=0, minf=0
  IO depths    : 1=25.1%, 2=74.9%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=832619,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=465MiB/s (487MB/s), 465MiB/s-465MiB/s (487MB/s-487MB/s), io=3252MiB (3410MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733602,
  "timestamp_ms" : 1666733602544,
  "time" : "Tue Oct 25 23:33:22 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2728185856,
        "io_kbytes" : 2664244,
        "bw_bytes" : 389685167,
        "bw" : 380551,
        "iops" : 95137.837452,
        "runtime" : 7001,
        "total_ios" : 666060,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2254,
          "max" : 101570,
          "mean" : 2532.068260,
          "stddev" : 388.305482,
          "N" : 666060
        },
        "clat_ns" : {
          "min" : 150,
          "max" : 157766,
          "mean" : 17905.455248,
          "stddev" : 3291.383779,
          "N" : 666061,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 17024,
            "30.000000" : 17024,
            "40.000000" : 17024,
            "50.000000" : 17280,
            "60.000000" : 17536,
            "70.000000" : 17536,
            "80.000000" : 17536,
            "90.000000" : 17792,
            "95.000000" : 19840,
            "99.000000" : 34048,
            "99.500000" : 40192,
            "99.900000" : 50432,
            "99.950000" : 63232,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11642,
          "max" : 160081,
          "mean" : 20437.523182,
          "stddev" : 3295.392638,
          "N" : 666061
        },
        "bw_min" : 379696,
        "bw_max" : 382128,
        "bw_agg" : 100.000000,
        "bw_mean" : 380707.500000,
        "bw_dev" : 705.250065,
        "bw_samples" : 14,
        "iops_min" : 94924,
        "iops_max" : 95532,
        "iops_mean" : 95176.857143,
        "iops_stddev" : 176.305180,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.700000,
      "sys_cpu" : 41.028571,
      "ctx" : 332854,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.016215,
        "20" : 95.018317,
        "50" : 4.859622,
        "100" : 0.102093,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 965358,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 12002,
      "write_ticks" : 0,
      "in_queue" : 12002,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86219: Tue Oct 25 23:33:22 2022
  read: IOPS=95.1k, BW=372MiB/s (390MB/s)(2602MiB/7001msec)
    slat (nsec): min=2254, max=101570, avg=2532.07, stdev=388.31
    clat (nsec): min=150, max=157766, avg=17905.46, stdev=3291.38
     lat (usec): min=11, max=160, avg=20.44, stdev= 3.30
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[16768], 10.00th=[16768], 20.00th=[17024],
     | 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17536],
     | 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19840],
     | 99.00th=[34048], 99.50th=[40192], 99.90th=[50432], 99.95th=[63232],
     | 99.99th=[74240]
   bw (  KiB/s): min=379696, max=382128, per=100.00%, avg=380707.50, stdev=705.25, samples=14
   iops        : min=94924, max=95532, avg=95176.86, stdev=176.31, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=0.02%, 20=95.02%, 50=4.86%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=12.70%, sys=41.03%, ctx=332854, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=666060,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=2602MiB (2728MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=965358/0, merge=0/0, ticks=12002/0, in_queue=12002, util=99.09%

$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_4K_2_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733616,
  "timestamp_ms" : 1666733616144,
  "time" : "Tue Oct 25 23:33:36 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2763063296,
        "io_kbytes" : 2698304,
        "bw_bytes" : 394666947,
        "bw" : 385416,
        "iops" : 96354.092273,
        "runtime" : 7001,
        "total_ios" : 674575,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2134,
          "max" : 30237,
          "mean" : 2411.264627,
          "stddev" : 301.369070,
          "N" : 674575
        },
        "clat_ns" : {
          "min" : 3416,
          "max" : 152055,
          "mean" : 17806.873363,
          "stddev" : 3321.266952,
          "N" : 674576,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 17024,
            "40.000000" : 17024,
            "50.000000" : 17280,
            "60.000000" : 17280,
            "70.000000" : 17536,
            "80.000000" : 17536,
            "90.000000" : 17536,
            "95.000000" : 20096,
            "99.000000" : 34048,
            "99.500000" : 40192,
            "99.900000" : 53504,
            "99.950000" : 65280,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 11702,
          "max" : 154269,
          "mean" : 20218.137700,
          "stddev" : 3319.032476,
          "N" : 674576
        },
        "bw_min" : 384640,
        "bw_max" : 386432,
        "bw_agg" : 100.000000,
        "bw_mean" : 385564.928571,
        "bw_dev" : 441.118241,
        "bw_samples" : 14,
        "iops_min" : 96160,
        "iops_max" : 96608,
        "iops_mean" : 96391.142857,
        "iops_stddev" : 110.190644,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.157143,
      "sys_cpu" : 41.785714,
      "ctx" : 337155,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 94.990624,
        "50" : 4.873884,
        "100" : 0.124226,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 966533,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11927,
      "write_ticks" : 0,
      "in_queue" : 11927,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86257: Tue Oct 25 23:33:36 2022
  read: IOPS=96.4k, BW=376MiB/s (395MB/s)(2635MiB/7001msec)
    slat (nsec): min=2134, max=30237, avg=2411.26, stdev=301.37
    clat (usec): min=3, max=152, avg=17.81, stdev= 3.32
     lat (usec): min=11, max=154, avg=20.22, stdev= 3.32
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17280],
     | 70.00th=[17536], 80.00th=[17536], 90.00th=[17536], 95.00th=[20096],
     | 99.00th=[34048], 99.50th=[40192], 99.90th=[53504], 99.95th=[65280],
     | 99.99th=[76288]
   bw (  KiB/s): min=384640, max=386432, per=100.00%, avg=385564.93, stdev=441.12, samples=14
   iops        : min=96160, max=96608, avg=96391.14, stdev=110.19, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=94.99%, 50=4.87%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=11.16%, sys=41.79%, ctx=337155, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=674575,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=376MiB/s (395MB/s), 376MiB/s-376MiB/s (395MB/s-395MB/s), io=2635MiB (2763MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=966533/0, merge=0/0, ticks=11927/0, in_queue=11927, util=99.09%

$ rm /tmp/fio-output_4K_2_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733629,
  "timestamp_ms" : 1666733629769,
  "time" : "Tue Oct 25 23:33:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2769817600,
        "io_kbytes" : 2704900,
        "bw_bytes" : 395631709,
        "bw" : 386359,
        "iops" : 96589.630053,
        "runtime" : 7001,
        "total_ios" : 676224,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2174,
          "max" : 56947,
          "mean" : 2455.820520,
          "stddev" : 313.047630,
          "N" : 676224
        },
        "clat_ns" : {
          "min" : 3426,
          "max" : 121939,
          "mean" : 17701.656814,
          "stddev" : 3232.617042,
          "N" : 676225,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 16768,
            "40.000000" : 16768,
            "50.000000" : 17280,
            "60.000000" : 17280,
            "70.000000" : 17280,
            "80.000000" : 17280,
            "90.000000" : 17536,
            "95.000000" : 19840,
            "99.000000" : 34048,
            "99.500000" : 36608,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 11822,
          "max" : 124183,
          "mean" : 20157.477096,
          "stddev" : 3230.430762,
          "N" : 676225
        },
        "bw_min" : 385176,
        "bw_max" : 387294,
        "bw_agg" : 100.000000,
        "bw_mean" : 386515.642857,
        "bw_dev" : 702.422359,
        "bw_samples" : 14,
        "iops_min" : 96294,
        "iops_max" : 96823,
        "iops_mean" : 96628.857143,
        "iops_stddev" : 175.563207,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.471429,
      "sys_cpu" : 42.771429,
      "ctx" : 337987,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 94.997516,
        "50" : 4.886694,
        "100" : 0.107509,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 967584,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11897,
      "write_ticks" : 0,
      "in_queue" : 11897,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=86295: Tue Oct 25 23:33:49 2022
  read: IOPS=96.6k, BW=377MiB/s (396MB/s)(2642MiB/7001msec)
    slat (nsec): min=2174, max=56947, avg=2455.82, stdev=313.05
    clat (usec): min=3, max=121, avg=17.70, stdev= 3.23
     lat (usec): min=11, max=124, avg=20.16, stdev= 3.23
    clat percentiles (nsec):
     |  1.00th=[16512],  5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[16768], 40.00th=[16768], 50.00th=[17280], 60.00th=[17280],
     | 70.00th=[17280], 80.00th=[17280], 90.00th=[17536], 95.00th=[19840],
     | 99.00th=[34048], 99.50th=[36608], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=385176, max=387294, per=100.00%, avg=386515.64, stdev=702.42, samples=14
   iops        : min=96294, max=96823, avg=96628.86, stdev=175.56, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=95.00%, 50=4.89%, 100=0.11%
  lat (usec)   : 250=0.01%
  cpu          : usr=10.47%, sys=42.77%, ctx=337987, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=676224,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=377MiB/s (396MB/s), 377MiB/s-377MiB/s (396MB/s-396MB/s), io=2642MiB (2770MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=967584/0, merge=0/0, ticks=11897/0, in_queue=11897, util=99.09%

$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733644,
  "timestamp_ms" : 1666733644623,
  "time" : "Tue Oct 25 23:34:04 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3403218944,
        "io_kbytes" : 3323456,
        "bw_bytes" : 486104691,
        "bw" : 474711,
        "iops" : 118677.760320,
        "runtime" : 7001,
        "total_ios" : 830863,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 13305,
          "mean" : 152.035344,
          "stddev" : 44.246764,
          "N" : 830863
        },
        "clat_ns" : {
          "min" : 11011,
          "max" : 125345,
          "mean" : 16229.591323,
          "stddev" : 3439.794272,
          "N" : 830863,
          "percentile" : {
            "1.000000" : 15168,
            "5.000000" : 15168,
            "10.000000" : 15296,
            "20.000000" : 15296,
            "30.000000" : 15424,
            "40.000000" : 15552,
            "50.000000" : 15552,
            "60.000000" : 15680,
            "70.000000" : 15680,
            "80.000000" : 15808,
            "90.000000" : 15936,
            "95.000000" : 16512,
            "99.000000" : 31872,
            "99.500000" : 37632,
            "99.900000" : 49920,
            "99.950000" : 66048,
            "99.990000" : 84480
          }
        },
        "lat_ns" : {
          "min" : 11191,
          "max" : 125525,
          "mean" : 16381.626666,
          "stddev" : 3441.514411,
          "N" : 830863
        },
        "bw_min" : 473656,
        "bw_max" : 476112,
        "bw_agg" : 100.000000,
        "bw_mean" : 474806.214286,
        "bw_dev" : 872.970718,
        "bw_samples" : 14,
        "iops_min" : 118414,
        "iops_max" : 119028,
        "iops_mean" : 118701.214286,
        "iops_stddev" : 218.461818,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 77.185714,
      "sys_cpu" : 22.785714,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.969128,
        "2" : 50.030872,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 95.630567,
        "50" : 4.272425,
        "100" : 0.096526,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86344: Tue Oct 25 23:34:04 2022
  read: IOPS=119k, BW=464MiB/s (486MB/s)(3246MiB/7001msec)
    slat (nsec): min=110, max=13305, avg=152.04, stdev=44.25
    clat (usec): min=11, max=125, avg=16.23, stdev= 3.44
     lat (usec): min=11, max=125, avg=16.38, stdev= 3.44
    clat percentiles (nsec):
     |  1.00th=[15168],  5.00th=[15168], 10.00th=[15296], 20.00th=[15296],
     | 30.00th=[15424], 40.00th=[15552], 50.00th=[15552], 60.00th=[15680],
     | 70.00th=[15680], 80.00th=[15808], 90.00th=[15936], 95.00th=[16512],
     | 99.00th=[31872], 99.50th=[37632], 99.90th=[49920], 99.95th=[66048],
     | 99.99th=[84480]
   bw (  KiB/s): min=473656, max=476112, per=100.00%, avg=474806.21, stdev=872.97, samples=14
   iops        : min=118414, max=119028, avg=118701.21, stdev=218.46, samples=14
  lat (usec)   : 20=95.63%, 50=4.27%, 100=0.10%, 250=0.01%
  cpu          : usr=77.19%, sys=22.79%, ctx=16, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=830863,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=464MiB/s (486MB/s), 464MiB/s-464MiB/s (486MB/s-486MB/s), io=3246MiB (3403MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt"
$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733659,
  "timestamp_ms" : 1666733659767,
  "time" : "Tue Oct 25 23:34:19 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3699757056,
        "io_kbytes" : 3613044,
        "bw_bytes" : 528461227,
        "bw" : 516075,
        "iops" : 129018.711613,
        "runtime" : 7001,
        "total_ios" : 903260,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 13215,
          "mean" : 159.408986,
          "stddev" : 56.703624,
          "N" : 903260
        },
        "clat_ns" : {
          "min" : 9217,
          "max" : 1316439,
          "mean" : 14838.287266,
          "stddev" : 4093.044611,
          "N" : 903260,
          "percentile" : {
            "1.000000" : 13888,
            "5.000000" : 13888,
            "10.000000" : 13888,
            "20.000000" : 14016,
            "30.000000" : 14016,
            "40.000000" : 14144,
            "50.000000" : 14144,
            "60.000000" : 14272,
            "70.000000" : 14272,
            "80.000000" : 14400,
            "90.000000" : 14400,
            "95.000000" : 15168,
            "99.000000" : 31616,
            "99.500000" : 39168,
            "99.900000" : 48384,
            "99.950000" : 66048,
            "99.990000" : 84480
          }
        },
        "lat_ns" : {
          "min" : 9407,
          "max" : 1316629,
          "mean" : 14997.696252,
          "stddev" : 4094.615509,
          "N" : 903260
        },
        "bw_min" : 514352,
        "bw_max" : 517672,
        "bw_agg" : 100.000000,
        "bw_mean" : 516195.357143,
        "bw_dev" : 856.680679,
        "bw_samples" : 14,
        "iops_min" : 128588,
        "iops_max" : 129418,
        "iops_mean" : 129048.785714,
        "iops_stddev" : 214.201264,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 68.600000,
      "sys_cpu" : 31.357143,
      "ctx" : 50,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.986936,
        "2" : 50.013064,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.020924,
        "20" : 96.007240,
        "50" : 3.875518,
        "100" : 0.095654,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86393: Tue Oct 25 23:34:19 2022
  read: IOPS=129k, BW=504MiB/s (528MB/s)(3528MiB/7001msec)
    slat (nsec): min=110, max=13215, avg=159.41, stdev=56.70
    clat (usec): min=9, max=1316, avg=14.84, stdev= 4.09
     lat (usec): min=9, max=1316, avg=15.00, stdev= 4.09
    clat percentiles (nsec):
     |  1.00th=[13888],  5.00th=[13888], 10.00th=[13888], 20.00th=[14016],
     | 30.00th=[14016], 40.00th=[14144], 50.00th=[14144], 60.00th=[14272],
     | 70.00th=[14272], 80.00th=[14400], 90.00th=[14400], 95.00th=[15168],
     | 99.00th=[31616], 99.50th=[39168], 99.90th=[48384], 99.95th=[66048],
     | 99.99th=[84480]
   bw (  KiB/s): min=514352, max=517672, per=100.00%, avg=516195.36, stdev=856.68, samples=14
   iops        : min=128588, max=129418, avg=129048.79, stdev=214.20, samples=14
  lat (usec)   : 10=0.02%, 20=96.01%, 50=3.88%, 100=0.10%, 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=68.60%, sys=31.36%, ctx=50, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=903260,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=504MiB/s (528MB/s), 504MiB/s-504MiB/s (528MB/s-528MB/s), io=3528MiB (3700MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733674,
  "timestamp_ms" : 1666733674905,
  "time" : "Tue Oct 25 23:34:34 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3672137728,
        "io_kbytes" : 3586072,
        "bw_bytes" : 524516173,
        "bw" : 512222,
        "iops" : 128055.563491,
        "runtime" : 7001,
        "total_ios" : 896517,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 100,
          "max" : 16821,
          "mean" : 151.655918,
          "stddev" : 54.676795,
          "N" : 896517
        },
        "clat_ns" : {
          "min" : 9408,
          "max" : 102191,
          "mean" : 14971.500404,
          "stddev" : 3599.731107,
          "N" : 896517,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 14016,
            "10.000000" : 14144,
            "20.000000" : 14144,
            "30.000000" : 14144,
            "40.000000" : 14272,
            "50.000000" : 14272,
            "60.000000" : 14400,
            "70.000000" : 14400,
            "80.000000" : 14400,
            "90.000000" : 14528,
            "95.000000" : 15296,
            "99.000000" : 31616,
            "99.500000" : 39168,
            "99.900000" : 53504,
            "99.950000" : 67072,
            "99.990000" : 82432
          }
        },
        "lat_ns" : {
          "min" : 9578,
          "max" : 102372,
          "mean" : 15123.156323,
          "stddev" : 3601.418454,
          "N" : 896517
        },
        "bw_min" : 511288,
        "bw_max" : 513200,
        "bw_agg" : 100.000000,
        "bw_mean" : 512352.071429,
        "bw_dev" : 676.281287,
        "bw_samples" : 14,
        "iops_min" : 127820,
        "iops_max" : 128300,
        "iops_mean" : 128088.000000,
        "iops_stddev" : 169.357156,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 64.900000,
      "sys_cpu" : 35.071429,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.996933,
        "2" : 50.003067,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 95.756801,
        "50" : 4.139018,
        "100" : 0.101950,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=86443: Tue Oct 25 23:34:34 2022
  read: IOPS=128k, BW=500MiB/s (525MB/s)(3502MiB/7001msec)
    slat (nsec): min=100, max=16821, avg=151.66, stdev=54.68
    clat (usec): min=9, max=102, avg=14.97, stdev= 3.60
     lat (usec): min=9, max=102, avg=15.12, stdev= 3.60
    clat percentiles (nsec):
     |  1.00th=[14016],  5.00th=[14016], 10.00th=[14144], 20.00th=[14144],
     | 30.00th=[14144], 40.00th=[14272], 50.00th=[14272], 60.00th=[14400],
     | 70.00th=[14400], 80.00th=[14400], 90.00th=[14528], 95.00th=[15296],
     | 99.00th=[31616], 99.50th=[39168], 99.90th=[53504], 99.95th=[67072],
     | 99.99th=[82432]
   bw (  KiB/s): min=511288, max=513200, per=100.00%, avg=512352.07, stdev=676.28, samples=14
   iops        : min=127820, max=128300, avg=128088.00, stdev=169.36, samples=14
  lat (usec)   : 10=0.01%, 20=95.76%, 50=4.14%, 100=0.10%, 250=0.01%
  cpu          : usr=64.90%, sys=35.07%, ctx=16, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=896517,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=500MiB/s (525MB/s), 500MiB/s-500MiB/s (525MB/s-525MB/s), io=3502MiB (3672MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733689,
  "timestamp_ms" : 1666733689991,
  "time" : "Tue Oct 25 23:34:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3760844800,
        "io_kbytes" : 3672700,
        "bw_bytes" : 537186801,
        "bw" : 524596,
        "iops" : 131148.978717,
        "runtime" : 7001,
        "total_ios" : 918174,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 15720,
          "mean" : 1814.378433,
          "stddev" : 122.167456,
          "N" : 918174
        },
        "clat_ns" : {
          "min" : 261,
          "max" : 95289,
          "mean" : 13203.321858,
          "stddev" : 4291.070630,
          "N" : 918175,
          "percentile" : {
            "1.000000" : 10048,
            "5.000000" : 10176,
            "10.000000" : 10176,
            "20.000000" : 10176,
            "30.000000" : 10304,
            "40.000000" : 10304,
            "50.000000" : 14528,
            "60.000000" : 14656,
            "70.000000" : 14656,
            "80.000000" : 14784,
            "90.000000" : 14912,
            "95.000000" : 15552,
            "99.000000" : 31616,
            "99.500000" : 37120,
            "99.900000" : 44800,
            "99.950000" : 62720,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 9618,
          "max" : 97092,
          "mean" : 15017.700203,
          "stddev" : 4292.850203,
          "N" : 918175
        },
        "bw_min" : 523296,
        "bw_max" : 526613,
        "bw_agg" : 100.000000,
        "bw_mean" : 524883.928571,
        "bw_dev" : 955.591357,
        "bw_samples" : 14,
        "iops_min" : 130824,
        "iops_max" : 131653,
        "iops_mean" : 131220.928571,
        "iops_stddev" : 238.846737,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.700000,
      "sys_cpu" : 88.271429,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.690174,
        "20" : 95.137632,
        "50" : 4.081253,
        "100" : 0.088110,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86494: Tue Oct 25 23:34:49 2022
  read: IOPS=131k, BW=512MiB/s (537MB/s)(3587MiB/7001msec)
    slat (nsec): min=1643, max=15720, avg=1814.38, stdev=122.17
    clat (nsec): min=261, max=95289, avg=13203.32, stdev=4291.07
     lat (nsec): min=9618, max=97092, avg=15017.70, stdev=4292.85
    clat percentiles (nsec):
     |  1.00th=[10048],  5.00th=[10176], 10.00th=[10176], 20.00th=[10176],
     | 30.00th=[10304], 40.00th=[10304], 50.00th=[14528], 60.00th=[14656],
     | 70.00th=[14656], 80.00th=[14784], 90.00th=[14912], 95.00th=[15552],
     | 99.00th=[31616], 99.50th=[37120], 99.90th=[44800], 99.95th=[62720],
     | 99.99th=[76288]
   bw (  KiB/s): min=523296, max=526613, per=100.00%, avg=524883.93, stdev=955.59, samples=14
   iops        : min=130824, max=131653, avg=131220.93, stdev=238.85, samples=14
  lat (nsec)   : 500=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.69%, 20=95.14%, 50=4.08%
  lat (usec)   : 100=0.09%
  cpu          : usr=11.70%, sys=88.27%, ctx=13, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=918174,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3587MiB (3761MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733057,
  "timestamp_ms" : 1666733057326,
  "time" : "Tue Oct 25 23:24:17 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1745297408,
        "io_kbytes" : 1704392,
        "bw_bytes" : 249292587,
        "bw" : 243449,
        "iops" : 60862.448222,
        "runtime" : 7001,
        "total_ios" : 426098,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2294,
          "max" : 44142,
          "mean" : 2645.758187,
          "stddev" : 205.940659,
          "N" : 426098
        },
        "clat_ns" : {
          "min" : 351,
          "max" : 105057,
          "mean" : 13250.876676,
          "stddev" : 2201.080057,
          "N" : 426098,
          "percentile" : {
            "1.000000" : 12096,
            "5.000000" : 12096,
            "10.000000" : 12096,
            "20.000000" : 12224,
            "30.000000" : 12352,
            "40.000000" : 13248,
            "50.000000" : 13376,
            "60.000000" : 13376,
            "70.000000" : 13504,
            "80.000000" : 13504,
            "90.000000" : 13504,
            "95.000000" : 13632,
            "99.000000" : 22912,
            "99.500000" : 29312,
            "99.900000" : 38656,
            "99.950000" : 52992,
            "99.990000" : 69120
          }
        },
        "lat_ns" : {
          "min" : 14187,
          "max" : 114945,
          "mean" : 15896.634863,
          "stddev" : 2223.085816,
          "N" : 426098
        },
        "bw_min" : 235512,
        "bw_max" : 257448,
        "bw_agg" : 100.000000,
        "bw_mean" : 243501.285714,
        "bw_dev" : 9024.250392,
        "bw_samples" : 14,
        "iops_min" : 58878,
        "iops_max" : 64362,
        "iops_mean" : 60875.285714,
        "iops_stddev" : 2256.088870,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.300000,
      "sys_cpu" : 34.514286,
      "ctx" : 426091,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.344982,
        "50" : 1.589306,
        "100" : 0.061488,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84545: Tue Oct 25 23:24:17 2022
  read: IOPS=60.9k, BW=238MiB/s (249MB/s)(1664MiB/7001msec)
    slat (nsec): min=2294, max=44142, avg=2645.76, stdev=205.94
    clat (nsec): min=351, max=105057, avg=13250.88, stdev=2201.08
     lat (usec): min=14, max=114, avg=15.90, stdev= 2.22
    clat percentiles (nsec):
     |  1.00th=[12096],  5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
     | 30.00th=[12352], 40.00th=[13248], 50.00th=[13376], 60.00th=[13376],
     | 70.00th=[13504], 80.00th=[13504], 90.00th=[13504], 95.00th=[13632],
     | 99.00th=[22912], 99.50th=[29312], 99.90th=[38656], 99.95th=[52992],
     | 99.99th=[69120]
   bw (  KiB/s): min=235512, max=257448, per=100.00%, avg=243501.29, stdev=9024.25, samples=14
   iops        : min=58878, max=64362, avg=60875.29, stdev=2256.09, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.01%, 20=98.34%, 50=1.59%, 100=0.06%, 250=0.01%
  cpu          : usr=9.30%, sys=34.51%, ctx=426091, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=426098,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=238MiB/s (249MB/s), 238MiB/s-238MiB/s (249MB/s-249MB/s), io=1664MiB (1745MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733705,
  "timestamp_ms" : 1666733705068,
  "time" : "Tue Oct 25 23:35:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3756642304,
        "io_kbytes" : 3668596,
        "bw_bytes" : 536586531,
        "bw" : 524010,
        "iops" : 131002.428225,
        "runtime" : 7001,
        "total_ios" : 917148,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 19396,
          "mean" : 1811.534450,
          "stddev" : 120.874810,
          "N" : 917148
        },
        "clat_ns" : {
          "min" : 591,
          "max" : 102302,
          "mean" : 13208.573923,
          "stddev" : 4374.320123,
          "N" : 917149,
          "percentile" : {
            "1.000000" : 10048,
            "5.000000" : 10176,
            "10.000000" : 10176,
            "20.000000" : 10176,
            "30.000000" : 10304,
            "40.000000" : 10304,
            "50.000000" : 14528,
            "60.000000" : 14656,
            "70.000000" : 14656,
            "80.000000" : 14784,
            "90.000000" : 14912,
            "95.000000" : 15552,
            "99.000000" : 31872,
            "99.500000" : 37120,
            "99.900000" : 50432,
            "99.950000" : 65280,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 9618,
          "max" : 104076,
          "mean" : 15020.108310,
          "stddev" : 4377.112921,
          "N" : 917149
        },
        "bw_min" : 522248,
        "bw_max" : 526256,
        "bw_agg" : 100.000000,
        "bw_mean" : 524297.500000,
        "bw_dev" : 1261.415673,
        "bw_samples" : 14,
        "iops_min" : 130562,
        "iops_max" : 131564,
        "iops_mean" : 131074.214286,
        "iops_stddev" : 315.322101,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.914286,
      "sys_cpu" : 89.071429,
      "ctx" : 17,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.643953,
        "20" : 95.156507,
        "50" : 4.090834,
        "100" : 0.104999,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86544: Tue Oct 25 23:35:05 2022
  read: IOPS=131k, BW=512MiB/s (537MB/s)(3583MiB/7001msec)
    slat (nsec): min=1643, max=19396, avg=1811.53, stdev=120.87
    clat (nsec): min=591, max=102302, avg=13208.57, stdev=4374.32
     lat (usec): min=9, max=104, avg=15.02, stdev= 4.38
    clat percentiles (nsec):
     |  1.00th=[10048],  5.00th=[10176], 10.00th=[10176], 20.00th=[10176],
     | 30.00th=[10304], 40.00th=[10304], 50.00th=[14528], 60.00th=[14656],
     | 70.00th=[14656], 80.00th=[14784], 90.00th=[14912], 95.00th=[15552],
     | 99.00th=[31872], 99.50th=[37120], 99.90th=[50432], 99.95th=[65280],
     | 99.99th=[77312]
   bw (  KiB/s): min=522248, max=526256, per=100.00%, avg=524297.50, stdev=1261.42, samples=14
   iops        : min=130562, max=131564, avg=131074.21, stdev=315.32, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.64%, 20=95.16%, 50=4.09%
  lat (usec)   : 100=0.10%, 250=0.01%
  cpu          : usr=10.91%, sys=89.07%, ctx=17, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=917148,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3583MiB (3757MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733720,
  "timestamp_ms" : 1666733720129,
  "time" : "Tue Oct 25 23:35:20 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3880730624,
        "io_kbytes" : 3789776,
        "bw_bytes" : 554310901,
        "bw" : 541319,
        "iops" : 135329.667190,
        "runtime" : 7001,
        "total_ios" : 947443,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1492,
          "max" : 15409,
          "mean" : 1579.608812,
          "stddev" : 127.933694,
          "N" : 947443
        },
        "clat_ns" : {
          "min" : 270,
          "max" : 118652,
          "mean" : 12956.637074,
          "stddev" : 3894.360504,
          "N" : 947444,
          "percentile" : {
            "1.000000" : 10048,
            "5.000000" : 10176,
            "10.000000" : 10176,
            "20.000000" : 11968,
            "30.000000" : 12096,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 14144,
            "90.000000" : 14400,
            "95.000000" : 14656,
            "99.000000" : 30592,
            "99.500000" : 37120,
            "99.900000" : 52480,
            "99.950000" : 64768,
            "99.990000" : 75264
          }
        },
        "lat_ns" : {
          "min" : 9178,
          "max" : 120225,
          "mean" : 14536.245858,
          "stddev" : 3892.786982,
          "N" : 947444
        },
        "bw_min" : 540352,
        "bw_max" : 542264,
        "bw_agg" : 100.000000,
        "bw_mean" : 541450.857143,
        "bw_dev" : 675.183719,
        "bw_samples" : 14,
        "iops_min" : 135088,
        "iops_max" : 135566,
        "iops_mean" : 135362.571429,
        "iops_stddev" : 168.735439,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.314286,
      "sys_cpu" : 90.671429,
      "ctx" : 7,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.089082,
        "20" : 96.324423,
        "50" : 3.473665,
        "100" : 0.109769,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=86593: Tue Oct 25 23:35:20 2022
  read: IOPS=135k, BW=529MiB/s (554MB/s)(3701MiB/7001msec)
    slat (nsec): min=1492, max=15409, avg=1579.61, stdev=127.93
    clat (nsec): min=270, max=118652, avg=12956.64, stdev=3894.36
     lat (usec): min=9, max=120, avg=14.54, stdev= 3.89
    clat percentiles (nsec):
     |  1.00th=[10048],  5.00th=[10176], 10.00th=[10176], 20.00th=[11968],
     | 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[14144], 90.00th=[14400], 95.00th=[14656],
     | 99.00th=[30592], 99.50th=[37120], 99.90th=[52480], 99.95th=[64768],
     | 99.99th=[75264]
   bw (  KiB/s): min=540352, max=542264, per=100.00%, avg=541450.86, stdev=675.18, samples=14
   iops        : min=135088, max=135566, avg=135362.57, stdev=168.74, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.09%, 20=96.32%, 50=3.47%, 100=0.11%
  lat (usec)   : 250=0.01%
  cpu          : usr=9.31%, sys=90.67%, ctx=7, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=947443,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=529MiB/s (554MB/s), 529MiB/s-529MiB/s (554MB/s-554MB/s), io=3701MiB (3881MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733733,
  "timestamp_ms" : 1666733733849,
  "time" : "Tue Oct 25 23:35:33 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2654441472,
        "io_kbytes" : 2592228,
        "bw_bytes" : 379151760,
        "bw" : 370265,
        "iops" : 92566.061991,
        "runtime" : 7001,
        "total_ios" : 648055,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2304,
          "max" : 769522,
          "mean" : 2713.917901,
          "stddev" : 997.462060,
          "N" : 648056
        },
        "clat_ns" : {
          "min" : 341,
          "max" : 775153,
          "mean" : 18398.808790,
          "stddev" : 3486.347443,
          "N" : 648056,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 16768,
            "10.000000" : 17024,
            "20.000000" : 17280,
            "30.000000" : 17536,
            "40.000000" : 17792,
            "50.000000" : 17792,
            "60.000000" : 18048,
            "70.000000" : 18048,
            "80.000000" : 18304,
            "90.000000" : 18304,
            "95.000000" : 20864,
            "99.000000" : 34560,
            "99.500000" : 43776,
            "99.900000" : 50432,
            "99.950000" : 63232,
            "99.990000" : 75264
          }
        },
        "lat_ns" : {
          "min" : 12253,
          "max" : 792535,
          "mean" : 21112.726690,
          "stddev" : 3619.369648,
          "N" : 648056
        },
        "bw_min" : 365080,
        "bw_max" : 384096,
        "bw_agg" : 100.000000,
        "bw_mean" : 370353.714286,
        "bw_dev" : 8054.404657,
        "bw_samples" : 14,
        "iops_min" : 91270,
        "iops_max" : 96024,
        "iops_mean" : 92588.428571,
        "iops_stddev" : 2013.601164,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.171429,
      "sys_cpu" : 40.400000,
      "ctx" : 323954,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 94.840870,
        "50" : 5.042628,
        "100" : 0.106010,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86632: Tue Oct 25 23:35:33 2022
  read: IOPS=92.6k, BW=362MiB/s (379MB/s)(2531MiB/7001msec)
    slat (nsec): min=2304, max=769522, avg=2713.92, stdev=997.46
    clat (nsec): min=341, max=775153, avg=18398.81, stdev=3486.35
     lat (usec): min=12, max=792, avg=21.11, stdev= 3.62
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[16768], 10.00th=[17024], 20.00th=[17280],
     | 30.00th=[17536], 40.00th=[17792], 50.00th=[17792], 60.00th=[18048],
     | 70.00th=[18048], 80.00th=[18304], 90.00th=[18304], 95.00th=[20864],
     | 99.00th=[34560], 99.50th=[43776], 99.90th=[50432], 99.95th=[63232],
     | 99.99th=[75264]
   bw (  KiB/s): min=365080, max=384096, per=100.00%, avg=370353.71, stdev=8054.40, samples=14
   iops        : min=91270, max=96024, avg=92588.43, stdev=2013.60, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=94.84%, 50=5.04%, 100=0.11%
  lat (usec)   : 250=0.01%, 1000=0.01%
  cpu          : usr=11.17%, sys=40.40%, ctx=323954, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=648055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=362MiB/s (379MB/s), 362MiB/s-362MiB/s (379MB/s-379MB/s), io=2531MiB (2654MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733747,
  "timestamp_ms" : 1666733747490,
  "time" : "Tue Oct 25 23:35:47 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2740174848,
        "io_kbytes" : 2675952,
        "bw_bytes" : 391397635,
        "bw" : 382224,
        "iops" : 95555.920583,
        "runtime" : 7001,
        "total_ios" : 668987,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2274,
          "max" : 30738,
          "mean" : 2583.744357,
          "stddev" : 358.833559,
          "N" : 668987
        },
        "clat_ns" : {
          "min" : 430,
          "max" : 156663,
          "mean" : 17860.045107,
          "stddev" : 3321.104405,
          "N" : 668988,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 17024,
            "30.000000" : 17024,
            "40.000000" : 17024,
            "50.000000" : 17280,
            "60.000000" : 17280,
            "70.000000" : 17536,
            "80.000000" : 17536,
            "90.000000" : 17792,
            "95.000000" : 19584,
            "99.000000" : 34048,
            "99.500000" : 40704,
            "99.900000" : 51456,
            "99.950000" : 64768,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 12112,
          "max" : 159028,
          "mean" : 20443.789060,
          "stddev" : 3321.208162,
          "N" : 668988
        },
        "bw_min" : 380880,
        "bw_max" : 384376,
        "bw_agg" : 100.000000,
        "bw_mean" : 382313.500000,
        "bw_dev" : 1209.400917,
        "bw_samples" : 14,
        "iops_min" : 95220,
        "iops_max" : 96094,
        "iops_mean" : 95578.357143,
        "iops_stddev" : 302.342548,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.242857,
      "sys_cpu" : 41.428571,
      "ctx" : 334337,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 95.022474,
        "50" : 4.850617,
        "100" : 0.116744,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86671: Tue Oct 25 23:35:47 2022
  read: IOPS=95.6k, BW=373MiB/s (391MB/s)(2613MiB/7001msec)
    slat (nsec): min=2274, max=30738, avg=2583.74, stdev=358.83
    clat (nsec): min=430, max=156663, avg=17860.05, stdev=3321.10
     lat (usec): min=12, max=159, avg=20.44, stdev= 3.32
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[16768], 10.00th=[16768], 20.00th=[17024],
     | 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17280],
     | 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19584],
     | 99.00th=[34048], 99.50th=[40704], 99.90th=[51456], 99.95th=[64768],
     | 99.99th=[76288]
   bw (  KiB/s): min=380880, max=384376, per=100.00%, avg=382313.50, stdev=1209.40, samples=14
   iops        : min=95220, max=96094, avg=95578.36, stdev=302.34, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=95.02%, 50=4.85%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.24%, sys=41.43%, ctx=334337, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=668987,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=373MiB/s (391MB/s), 373MiB/s-373MiB/s (391MB/s-391MB/s), io=2613MiB (2740MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_4K_2_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733761,
  "timestamp_ms" : 1666733761130,
  "time" : "Tue Oct 25 23:36:01 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2729132032,
        "io_kbytes" : 2665168,
        "bw_bytes" : 389820315,
        "bw" : 380683,
        "iops" : 95170.689901,
        "runtime" : 7001,
        "total_ios" : 666290,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2294,
          "max" : 39635,
          "mean" : 2615.013979,
          "stddev" : 376.495980,
          "N" : 666291
        },
        "clat_ns" : {
          "min" : 370,
          "max" : 191489,
          "mean" : 17882.856067,
          "stddev" : 3375.558001,
          "N" : 666291,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 16768,
            "30.000000" : 17024,
            "40.000000" : 17024,
            "50.000000" : 17280,
            "60.000000" : 17536,
            "70.000000" : 17536,
            "80.000000" : 17536,
            "90.000000" : 17792,
            "95.000000" : 19584,
            "99.000000" : 34048,
            "99.500000" : 43264,
            "99.900000" : 53504,
            "99.950000" : 65280,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 11221,
          "max" : 193854,
          "mean" : 20497.870046,
          "stddev" : 3378.217884,
          "N" : 666291
        },
        "bw_min" : 378240,
        "bw_max" : 382600,
        "bw_agg" : 100.000000,
        "bw_mean" : 380772.071429,
        "bw_dev" : 1090.263798,
        "bw_samples" : 14,
        "iops_min" : 94560,
        "iops_max" : 95650,
        "iops_mean" : 95193.000000,
        "iops_stddev" : 272.562484,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.257143,
      "sys_cpu" : 39.942857,
      "ctx" : 332996,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 95.048703,
        "50" : 4.824326,
        "100" : 0.118717,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=86711: Tue Oct 25 23:36:01 2022
  read: IOPS=95.2k, BW=372MiB/s (390MB/s)(2603MiB/7001msec)
    slat (nsec): min=2294, max=39635, avg=2615.01, stdev=376.50
    clat (nsec): min=370, max=191489, avg=17882.86, stdev=3375.56
     lat (usec): min=11, max=193, avg=20.50, stdev= 3.38
    clat percentiles (nsec):
     |  1.00th=[16768],  5.00th=[16768], 10.00th=[16768], 20.00th=[16768],
     | 30.00th=[17024], 40.00th=[17024], 50.00th=[17280], 60.00th=[17536],
     | 70.00th=[17536], 80.00th=[17536], 90.00th=[17792], 95.00th=[19584],
     | 99.00th=[34048], 99.50th=[43264], 99.90th=[53504], 99.95th=[65280],
     | 99.99th=[76288]
   bw (  KiB/s): min=378240, max=382600, per=100.00%, avg=380772.07, stdev=1090.26, samples=14
   iops        : min=94560, max=95650, avg=95193.00, stdev=272.56, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=95.05%, 50=4.82%, 100=0.12%
  lat (usec)   : 250=0.01%
  cpu          : usr=15.26%, sys=39.94%, ctx=332996, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=666290,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=372MiB/s (390MB/s), 372MiB/s-372MiB/s (390MB/s-390MB/s), io=2603MiB (2729MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733774,
  "timestamp_ms" : 1666733774795,
  "time" : "Tue Oct 25 23:36:14 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2756984832,
        "io_kbytes" : 2692368,
        "bw_bytes" : 393798719,
        "bw" : 384569,
        "iops" : 96142.122554,
        "runtime" : 7001,
        "total_ios" : 673091,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2394,
          "max" : 29245,
          "mean" : 2705.547284,
          "stddev" : 335.582093,
          "N" : 673091
        },
        "clat_ns" : {
          "min" : 802,
          "max" : 199354,
          "mean" : 17555.780315,
          "stddev" : 5106.894427,
          "N" : 673092,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 12864,
            "10.000000" : 12864,
            "20.000000" : 12864,
            "30.000000" : 12992,
            "40.000000" : 13120,
            "50.000000" : 20608,
            "60.000000" : 20864,
            "70.000000" : 20864,
            "80.000000" : 20864,
            "90.000000" : 21120,
            "95.000000" : 21632,
            "99.000000" : 33024,
            "99.500000" : 38656,
            "99.900000" : 48896,
            "99.950000" : 61696,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 11712,
          "max" : 202269,
          "mean" : 20261.327762,
          "stddev" : 5287.935425,
          "N" : 673092
        },
        "bw_min" : 383832,
        "bw_max" : 385720,
        "bw_agg" : 100.000000,
        "bw_mean" : 384720.071429,
        "bw_dev" : 610.691722,
        "bw_samples" : 14,
        "iops_min" : 95958,
        "iops_max" : 96430,
        "iops_mean" : 96179.928571,
        "iops_stddev" : 152.704070,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.714286,
      "sys_cpu" : 42.514286,
      "ctx" : 336367,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 47.780909,
        "50" : 52.115241,
        "100" : 0.093895,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 963565,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11496,
      "write_ticks" : 0,
      "in_queue" : 11496,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=86751: Tue Oct 25 23:36:14 2022
  read: IOPS=96.1k, BW=376MiB/s (394MB/s)(2629MiB/7001msec)
    slat (nsec): min=2394, max=29245, avg=2705.55, stdev=335.58
    clat (nsec): min=802, max=199354, avg=17555.78, stdev=5106.89
     lat (usec): min=11, max=202, avg=20.26, stdev= 5.29
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12864], 10.00th=[12864], 20.00th=[12864],
     | 30.00th=[12992], 40.00th=[13120], 50.00th=[20608], 60.00th=[20864],
     | 70.00th=[20864], 80.00th=[20864], 90.00th=[21120], 95.00th=[21632],
     | 99.00th=[33024], 99.50th=[38656], 99.90th=[48896], 99.95th=[61696],
     | 99.99th=[76288]
   bw (  KiB/s): min=383832, max=385720, per=100.00%, avg=384720.07, stdev=610.69, samples=14
   iops        : min=95958, max=96430, avg=96179.93, stdev=152.70, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=47.78%, 50=52.12%, 100=0.09%
  lat (usec)   : 250=0.01%
  cpu          : usr=14.71%, sys=42.51%, ctx=336367, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=673091,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=376MiB/s (394MB/s), 376MiB/s-376MiB/s (394MB/s-394MB/s), io=2629MiB (2757MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=963565/0, merge=0/0, ticks=11496/0, in_queue=11496, util=99.09%

$ rm /tmp/fio-output_4K_2_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733788,
  "timestamp_ms" : 1666733788456,
  "time" : "Tue Oct 25 23:36:28 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2764947456,
        "io_kbytes" : 2700144,
        "bw_bytes" : 394936074,
        "bw" : 385679,
        "iops" : 96419.654335,
        "runtime" : 7001,
        "total_ios" : 675034,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2384,
          "max" : 46998,
          "mean" : 2698.317754,
          "stddev" : 340.659642,
          "N" : 675035
        },
        "clat_ns" : {
          "min" : 882,
          "max" : 136166,
          "mean" : 17524.151786,
          "stddev" : 5078.250149,
          "N" : 675035,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 12864,
            "10.000000" : 12864,
            "20.000000" : 12992,
            "30.000000" : 12992,
            "40.000000" : 13120,
            "50.000000" : 20608,
            "60.000000" : 20608,
            "70.000000" : 20864,
            "80.000000" : 20864,
            "90.000000" : 20864,
            "95.000000" : 21632,
            "99.000000" : 33024,
            "99.500000" : 39168,
            "99.900000" : 52480,
            "99.950000" : 66048,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 12653,
          "max" : 139051,
          "mean" : 20222.470281,
          "stddev" : 5270.579236,
          "N" : 675035
        },
        "bw_min" : 385120,
        "bw_max" : 386536,
        "bw_agg" : 100.000000,
        "bw_mean" : 385827.285714,
        "bw_dev" : 381.399332,
        "bw_samples" : 14,
        "iops_min" : 96280,
        "iops_max" : 96634,
        "iops_mean" : 96456.714286,
        "iops_stddev" : 95.320051,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.000000,
      "sys_cpu" : 43.514286,
      "ctx" : 337349,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 47.800703,
        "50" : 52.086265,
        "100" : 0.105328,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 965663,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11565,
      "write_ticks" : 0,
      "in_queue" : 11565,
      "util" : 99.049128
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=86789: Tue Oct 25 23:36:28 2022
  read: IOPS=96.4k, BW=377MiB/s (395MB/s)(2637MiB/7001msec)
    slat (nsec): min=2384, max=46998, avg=2698.32, stdev=340.66
    clat (nsec): min=882, max=136166, avg=17524.15, stdev=5078.25
     lat (usec): min=12, max=139, avg=20.22, stdev= 5.27
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12864], 10.00th=[12864], 20.00th=[12992],
     | 30.00th=[12992], 40.00th=[13120], 50.00th=[20608], 60.00th=[20608],
     | 70.00th=[20864], 80.00th=[20864], 90.00th=[20864], 95.00th=[21632],
     | 99.00th=[33024], 99.50th=[39168], 99.90th=[52480], 99.95th=[66048],
     | 99.99th=[77312]
   bw (  KiB/s): min=385120, max=386536, per=100.00%, avg=385827.29, stdev=381.40, samples=14
   iops        : min=96280, max=96634, avg=96456.71, stdev=95.32, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=47.80%, 50=52.09%, 100=0.11%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.00%, sys=43.51%, ctx=337349, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=675034,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=377MiB/s (395MB/s), 377MiB/s-377MiB/s (395MB/s-395MB/s), io=2637MiB (2765MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=965663/0, merge=0/0, ticks=11565/0, in_queue=11565, util=99.05%

$ rm /tmp/fio-output_4K_2_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt"
$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_2_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733802,
  "timestamp_ms" : 1666733802098,
  "time" : "Tue Oct 25 23:36:42 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2776817664,
        "io_kbytes" : 2711736,
        "bw_bytes" : 396631576,
        "bw" : 387335,
        "iops" : 96833.595201,
        "runtime" : 7001,
        "total_ios" : 677932,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2374,
          "max" : 177503,
          "mean" : 2666.347233,
          "stddev" : 383.992040,
          "N" : 677933
        },
        "clat_ns" : {
          "min" : 822,
          "max" : 195737,
          "mean" : 17461.917285,
          "stddev" : 5079.205885,
          "N" : 677933,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 12736,
            "10.000000" : 12864,
            "20.000000" : 12864,
            "30.000000" : 12992,
            "40.000000" : 13120,
            "50.000000" : 20352,
            "60.000000" : 20608,
            "70.000000" : 20608,
            "80.000000" : 20864,
            "90.000000" : 20864,
            "95.000000" : 21632,
            "99.000000" : 33024,
            "99.500000" : 38144,
            "99.900000" : 50944,
            "99.950000" : 65280,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 13045,
          "max" : 198572,
          "mean" : 20128.265079,
          "stddev" : 5254.394288,
          "N" : 677933
        },
        "bw_min" : 385704,
        "bw_max" : 389835,
        "bw_agg" : 100.000000,
        "bw_mean" : 387489.428571,
        "bw_dev" : 1265.535503,
        "bw_samples" : 14,
        "iops_min" : 96426,
        "iops_max" : 97458,
        "iops_mean" : 96872.285714,
        "iops_stddev" : 316.254870,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.857143,
      "sys_cpu" : 41.557143,
      "ctx" : 338781,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 47.810990,
        "50" : 52.078527,
        "100" : 0.101043,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 970953,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 11596,
      "write_ticks" : 0,
      "in_queue" : 11596,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=86827: Tue Oct 25 23:36:42 2022
  read: IOPS=96.8k, BW=378MiB/s (397MB/s)(2648MiB/7001msec)
    slat (nsec): min=2374, max=177503, avg=2666.35, stdev=383.99
    clat (nsec): min=822, max=195737, avg=17461.92, stdev=5079.21
     lat (usec): min=13, max=198, avg=20.13, stdev= 5.25
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12736], 10.00th=[12864], 20.00th=[12864],
     | 30.00th=[12992], 40.00th=[13120], 50.00th=[20352], 60.00th=[20608],
     | 70.00th=[20608], 80.00th=[20864], 90.00th=[20864], 95.00th=[21632],
     | 99.00th=[33024], 99.50th=[38144], 99.90th=[50944], 99.95th=[65280],
     | 99.99th=[77312]
   bw (  KiB/s): min=385704, max=389835, per=100.00%, avg=387489.43, stdev=1265.54, samples=14
   iops        : min=96426, max=97458, avg=96872.29, stdev=316.25, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 10=0.01%, 20=47.81%, 50=52.08%, 100=0.10%, 250=0.01%
  cpu          : usr=14.86%, sys=41.56%, ctx=338781, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=677932,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=378MiB/s (397MB/s), 378MiB/s-378MiB/s (397MB/s-397MB/s), io=2648MiB (2777MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=970953/0, merge=0/0, ticks=11596/0, in_queue=11596, util=99.09%

$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733817,
  "timestamp_ms" : 1666733817120,
  "time" : "Tue Oct 25 23:36:57 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3817984000,
        "io_kbytes" : 3728500,
        "bw_bytes" : 545348378,
        "bw" : 532566,
        "iops" : 133141.551207,
        "runtime" : 7001,
        "total_ios" : 932124,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1503,
          "max" : 15279,
          "mean" : 1580.962068,
          "stddev" : 125.314578,
          "N" : 932125
        },
        "clat_ns" : {
          "min" : 261,
          "max" : 116468,
          "mean" : 12250.310121,
          "stddev" : 3714.120484,
          "N" : 932125,
          "percentile" : {
            "1.000000" : 10560,
            "5.000000" : 10560,
            "10.000000" : 10560,
            "20.000000" : 10688,
            "30.000000" : 10688,
            "40.000000" : 10816,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12480,
            "90.000000" : 12608,
            "95.000000" : 13120,
            "99.000000" : 29568,
            "99.500000" : 37120,
            "99.900000" : 45312,
            "99.950000" : 63232,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 9247,
          "max" : 118081,
          "mean" : 13831.272190,
          "stddev" : 3720.542513,
          "N" : 932125
        },
        "bw_min" : 531104,
        "bw_max" : 534640,
        "bw_agg" : 100.000000,
        "bw_mean" : 532689.428571,
        "bw_dev" : 1101.338606,
        "bw_samples" : 14,
        "iops_min" : 132776,
        "iops_max" : 133660,
        "iops_mean" : 133172.214286,
        "iops_stddev" : 275.363306,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 84.400000,
      "sys_cpu" : 15.585714,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.981869,
        "2" : 50.018131,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.049886,
        "20" : 96.550674,
        "50" : 3.300312,
        "100" : 0.096232,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86875: Tue Oct 25 23:36:57 2022
  read: IOPS=133k, BW=520MiB/s (545MB/s)(3641MiB/7001msec)
    slat (nsec): min=1503, max=15279, avg=1580.96, stdev=125.31
    clat (nsec): min=261, max=116468, avg=12250.31, stdev=3714.12
     lat (usec): min=9, max=118, avg=13.83, stdev= 3.72
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[10560], 10.00th=[10560], 20.00th=[10688],
     | 30.00th=[10688], 40.00th=[10816], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12480], 90.00th=[12608], 95.00th=[13120],
     | 99.00th=[29568], 99.50th=[37120], 99.90th=[45312], 99.95th=[63232],
     | 99.99th=[74240]
   bw (  KiB/s): min=531104, max=534640, per=100.00%, avg=532689.43, stdev=1101.34, samples=14
   iops        : min=132776, max=133660, avg=133172.21, stdev=275.36, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.05%, 20=96.55%, 50=3.30%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=84.40%, sys=15.59%, ctx=4, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=932124,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=520MiB/s (545MB/s), 520MiB/s-520MiB/s (545MB/s-545MB/s), io=3641MiB (3818MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733832,
  "timestamp_ms" : 1666733832158,
  "time" : "Tue Oct 25 23:37:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3812896768,
        "io_kbytes" : 3723532,
        "bw_bytes" : 544621735,
        "bw" : 531857,
        "iops" : 132964.147979,
        "runtime" : 7001,
        "total_ios" : 930882,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1542,
          "max" : 43872,
          "mean" : 1620.137457,
          "stddev" : 131.643678,
          "N" : 930882
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 1034130,
          "mean" : 12198.356105,
          "stddev" : 4029.463553,
          "N" : 930882,
          "percentile" : {
            "1.000000" : 10432,
            "5.000000" : 10432,
            "10.000000" : 10560,
            "20.000000" : 10560,
            "30.000000" : 10688,
            "40.000000" : 10688,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12480,
            "90.000000" : 12480,
            "95.000000" : 13120,
            "99.000000" : 29568,
            "99.500000" : 37120,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 9217,
          "max" : 1035763,
          "mean" : 13818.493562,
          "stddev" : 4036.311352,
          "N" : 930882
        },
        "bw_min" : 528936,
        "bw_max" : 533304,
        "bw_agg" : 100.000000,
        "bw_mean" : 531980.714286,
        "bw_dev" : 1458.161879,
        "bw_samples" : 14,
        "iops_min" : 132234,
        "iops_max" : 133326,
        "iops_mean" : 132995.142857,
        "iops_stddev" : 364.505747,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 86.657143,
      "sys_cpu" : 13.300000,
      "ctx" : 41,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.984746,
        "2" : 50.015254,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.049738,
        "20" : 96.555847,
        "50" : 3.289031,
        "100" : 0.102483,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86924: Tue Oct 25 23:37:12 2022
  read: IOPS=133k, BW=519MiB/s (545MB/s)(3636MiB/7001msec)
    slat (nsec): min=1542, max=43872, avg=1620.14, stdev=131.64
    clat (nsec): min=240, max=1034.1k, avg=12198.36, stdev=4029.46
     lat (usec): min=9, max=1035, avg=13.82, stdev= 4.04
    clat percentiles (nsec):
     |  1.00th=[10432],  5.00th=[10432], 10.00th=[10560], 20.00th=[10560],
     | 30.00th=[10688], 40.00th=[10688], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12480], 90.00th=[12480], 95.00th=[13120],
     | 99.00th=[29568], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=528936, max=533304, per=100.00%, avg=531980.71, stdev=1458.16, samples=14
   iops        : min=132234, max=133326, avg=132995.14, stdev=364.51, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.05%, 20=96.56%, 50=3.29%, 100=0.10%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=86.66%, sys=13.30%, ctx=41, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=930882,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=3636MiB (3813MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733070,
  "timestamp_ms" : 1666733070951,
  "time" : "Tue Oct 25 23:24:30 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 1847742464,
        "io_kbytes" : 1804436,
        "bw_bytes" : 263925505,
        "bw" : 257739,
        "iops" : 64434.937866,
        "runtime" : 7001,
        "total_ios" : 451109,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2314,
          "max" : 18234,
          "mean" : 2506.600283,
          "stddev" : 209.404360,
          "N" : 451109
        },
        "clat_ns" : {
          "min" : 341,
          "max" : 183043,
          "mean" : 12344.901031,
          "stddev" : 2052.638325,
          "N" : 451109,
          "percentile" : {
            "1.000000" : 11840,
            "5.000000" : 11840,
            "10.000000" : 11840,
            "20.000000" : 11968,
            "30.000000" : 11968,
            "40.000000" : 11968,
            "50.000000" : 12096,
            "60.000000" : 12096,
            "70.000000" : 12224,
            "80.000000" : 12224,
            "90.000000" : 12352,
            "95.000000" : 12480,
            "99.000000" : 22144,
            "99.500000" : 24448,
            "99.900000" : 31616,
            "99.950000" : 49920,
            "99.990000" : 64256
          }
        },
        "lat_ns" : {
          "min" : 11482,
          "max" : 185648,
          "mean" : 14851.501313,
          "stddev" : 2066.668313,
          "N" : 451109
        },
        "bw_min" : 257312,
        "bw_max" : 258252,
        "bw_agg" : 100.000000,
        "bw_mean" : 257799.142857,
        "bw_dev" : 320.139118,
        "bw_samples" : 14,
        "iops_min" : 64328,
        "iops_max" : 64563,
        "iops_mean" : 64449.785714,
        "iops_stddev" : 80.034780,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.671429,
      "sys_cpu" : 36.157143,
      "ctx" : 451088,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.461569,
        "50" : 1.477913,
        "100" : 0.051650,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=84584: Tue Oct 25 23:24:30 2022
  read: IOPS=64.4k, BW=252MiB/s (264MB/s)(1762MiB/7001msec)
    slat (nsec): min=2314, max=18234, avg=2506.60, stdev=209.40
    clat (nsec): min=341, max=183043, avg=12344.90, stdev=2052.64
     lat (usec): min=11, max=185, avg=14.85, stdev= 2.07
    clat percentiles (nsec):
     |  1.00th=[11840],  5.00th=[11840], 10.00th=[11840], 20.00th=[11968],
     | 30.00th=[11968], 40.00th=[11968], 50.00th=[12096], 60.00th=[12096],
     | 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12480],
     | 99.00th=[22144], 99.50th=[24448], 99.90th=[31616], 99.95th=[49920],
     | 99.99th=[64256]
   bw (  KiB/s): min=257312, max=258252, per=100.00%, avg=257799.14, stdev=320.14, samples=14
   iops        : min=64328, max=64563, avg=64449.79, stdev=80.03, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.46%, 50=1.48%, 100=0.05%
  lat (usec)   : 250=0.01%
  cpu          : usr=10.67%, sys=36.16%, ctx=451088, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=451109,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=252MiB/s (264MB/s), 252MiB/s-252MiB/s (264MB/s-264MB/s), io=1762MiB (1848MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_2_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733847,
  "timestamp_ms" : 1666733847292,
  "time" : "Tue Oct 25 23:37:27 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3812159488,
        "io_kbytes" : 3722812,
        "bw_bytes" : 544516424,
        "bw" : 531754,
        "iops" : 132938.437366,
        "runtime" : 7001,
        "total_ios" : 930702,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1542,
          "max" : 16170,
          "mean" : 1619.645983,
          "stddev" : 121.460440,
          "N" : 930702
        },
        "clat_ns" : {
          "min" : 251,
          "max" : 176772,
          "mean" : 12194.889915,
          "stddev" : 3779.824468,
          "N" : 930702,
          "percentile" : {
            "1.000000" : 10432,
            "5.000000" : 10560,
            "10.000000" : 10560,
            "20.000000" : 10560,
            "30.000000" : 10688,
            "40.000000" : 10688,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12480,
            "90.000000" : 12480,
            "95.000000" : 12992,
            "99.000000" : 29568,
            "99.500000" : 37120,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 74240
          }
        },
        "lat_ns" : {
          "min" : 9257,
          "max" : 178395,
          "mean" : 13814.535899,
          "stddev" : 3787.101803,
          "N" : 930702
        },
        "bw_min" : 530488,
        "bw_max" : 532609,
        "bw_agg" : 100.000000,
        "bw_mean" : 531886.928571,
        "bw_dev" : 581.030446,
        "bw_samples" : 14,
        "iops_min" : 132622,
        "iops_max" : 133152,
        "iops_mean" : 132971.714286,
        "iops_stddev" : 145.339617,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 84.900000,
      "sys_cpu" : 15.071429,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.984635,
        "2" : 50.015365,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.048458,
        "20" : 96.556255,
        "50" : 3.289452,
        "100" : 0.102288,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=86975: Tue Oct 25 23:37:27 2022
  read: IOPS=133k, BW=519MiB/s (545MB/s)(3636MiB/7001msec)
    slat (nsec): min=1542, max=16170, avg=1619.65, stdev=121.46
    clat (nsec): min=251, max=176772, avg=12194.89, stdev=3779.82
     lat (usec): min=9, max=178, avg=13.81, stdev= 3.79
    clat percentiles (nsec):
     |  1.00th=[10432],  5.00th=[10560], 10.00th=[10560], 20.00th=[10560],
     | 30.00th=[10688], 40.00th=[10688], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12480], 90.00th=[12480], 95.00th=[12992],
     | 99.00th=[29568], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[74240]
   bw (  KiB/s): min=530488, max=532609, per=100.00%, avg=531886.93, stdev=581.03, samples=14
   iops        : min=132622, max=133152, avg=132971.71, stdev=145.34, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.05%, 20=96.56%, 50=3.29%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=84.90%, sys=15.07%, ctx=16, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=930702,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=519MiB/s (545MB/s), 519MiB/s-519MiB/s (545MB/s-545MB/s), io=3636MiB (3812MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733862,
  "timestamp_ms" : 1666733862396,
  "time" : "Tue Oct 25 23:37:42 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3758456832,
        "io_kbytes" : 3670368,
        "bw_bytes" : 536845712,
        "bw" : 524263,
        "iops" : 131065.704899,
        "runtime" : 7001,
        "total_ios" : 917591,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1542,
          "max" : 1290761,
          "mean" : 1616.681889,
          "stddev" : 1355.420376,
          "N" : 917592
        },
        "clat_ns" : {
          "min" : 691,
          "max" : 1291843,
          "mean" : 12424.666600,
          "stddev" : 3999.828740,
          "N" : 917592,
          "percentile" : {
            "1.000000" : 10560,
            "5.000000" : 10560,
            "10.000000" : 10688,
            "20.000000" : 10816,
            "30.000000" : 10944,
            "40.000000" : 10944,
            "50.000000" : 12352,
            "60.000000" : 12480,
            "70.000000" : 12608,
            "80.000000" : 12736,
            "90.000000" : 12736,
            "95.000000" : 13376,
            "99.000000" : 29824,
            "99.500000" : 37120,
            "99.900000" : 50944,
            "99.950000" : 63744,
            "99.990000" : 75264
          }
        },
        "lat_ns" : {
          "min" : 9447,
          "max" : 1294137,
          "mean" : 14041.348489,
          "stddev" : 4225.469205,
          "N" : 917592
        },
        "bw_min" : 523344,
        "bw_max" : 524897,
        "bw_agg" : 100.000000,
        "bw_mean" : 524383.500000,
        "bw_dev" : 486.058995,
        "bw_samples" : 14,
        "iops_min" : 130836,
        "iops_max" : 131224,
        "iops_mean" : 131095.857143,
        "iops_stddev" : 121.494449,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 32.271429,
      "sys_cpu" : 67.685714,
      "ctx" : 43,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.985451,
        "2" : 50.014549,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.042394,
        "20" : 96.514896,
        "50" : 3.329261,
        "100" : 0.110616,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87024: Tue Oct 25 23:37:42 2022
  read: IOPS=131k, BW=512MiB/s (537MB/s)(3584MiB/7001msec)
    slat (nsec): min=1542, max=1290.8k, avg=1616.68, stdev=1355.42
    clat (nsec): min=691, max=1291.8k, avg=12424.67, stdev=3999.83
     lat (usec): min=9, max=1294, avg=14.04, stdev= 4.23
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[10560], 10.00th=[10688], 20.00th=[10816],
     | 30.00th=[10944], 40.00th=[10944], 50.00th=[12352], 60.00th=[12480],
     | 70.00th=[12608], 80.00th=[12736], 90.00th=[12736], 95.00th=[13376],
     | 99.00th=[29824], 99.50th=[37120], 99.90th=[50944], 99.95th=[63744],
     | 99.99th=[75264]
   bw (  KiB/s): min=523344, max=524897, per=100.00%, avg=524383.50, stdev=486.06, samples=14
   iops        : min=130836, max=131224, avg=131095.86, stdev=121.49, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.04%, 20=96.51%, 50=3.33%, 100=0.11%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=32.27%, sys=67.69%, ctx=43, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=917591,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3584MiB (3758MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733877,
  "timestamp_ms" : 1666733877524,
  "time" : "Tue Oct 25 23:37:57 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3756961792,
        "io_kbytes" : 3668908,
        "bw_bytes" : 536632165,
        "bw" : 524054,
        "iops" : 131013.569490,
        "runtime" : 7001,
        "total_ios" : 917226,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1543,
          "max" : 16892,
          "mean" : 1635.534015,
          "stddev" : 122.054841,
          "N" : 917227
        },
        "clat_ns" : {
          "min" : 711,
          "max" : 1299107,
          "mean" : 12389.945191,
          "stddev" : 4132.412283,
          "N" : 917227,
          "percentile" : {
            "1.000000" : 10560,
            "5.000000" : 10688,
            "10.000000" : 10688,
            "20.000000" : 10688,
            "30.000000" : 10816,
            "40.000000" : 11072,
            "50.000000" : 12352,
            "60.000000" : 12480,
            "70.000000" : 12480,
            "80.000000" : 12608,
            "90.000000" : 12864,
            "95.000000" : 13376,
            "99.000000" : 29568,
            "99.500000" : 36608,
            "99.900000" : 42240,
            "99.950000" : 62208,
            "99.990000" : 75264
          }
        },
        "lat_ns" : {
          "min" : 9368,
          "max" : 1300770,
          "mean" : 14025.479206,
          "stddev" : 4142.473008,
          "N" : 917227
        },
        "bw_min" : 523160,
        "bw_max" : 525571,
        "bw_agg" : 100.000000,
        "bw_mean" : 524175.642857,
        "bw_dev" : 851.873917,
        "bw_samples" : 14,
        "iops_min" : 130790,
        "iops_max" : 131392,
        "iops_mean" : 131043.857143,
        "iops_stddev" : 212.874053,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 35.357143,
      "sys_cpu" : 64.614286,
      "ctx" : 40,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.978522,
        "2" : 50.021478,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.056911,
        "20" : 96.554502,
        "50" : 3.304638,
        "100" : 0.081332,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87073: Tue Oct 25 23:37:57 2022
  read: IOPS=131k, BW=512MiB/s (537MB/s)(3583MiB/7001msec)
    slat (nsec): min=1543, max=16892, avg=1635.53, stdev=122.05
    clat (nsec): min=711, max=1299.1k, avg=12389.95, stdev=4132.41
     lat (usec): min=9, max=1300, avg=14.03, stdev= 4.14
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[10688], 10.00th=[10688], 20.00th=[10688],
     | 30.00th=[10816], 40.00th=[11072], 50.00th=[12352], 60.00th=[12480],
     | 70.00th=[12480], 80.00th=[12608], 90.00th=[12864], 95.00th=[13376],
     | 99.00th=[29568], 99.50th=[36608], 99.90th=[42240], 99.95th=[62208],
     | 99.99th=[75264]
   bw (  KiB/s): min=523160, max=525571, per=100.00%, avg=524175.64, stdev=851.87, samples=14
   iops        : min=130790, max=131392, avg=131043.86, stdev=212.87, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.06%, 20=96.55%, 50=3.30%, 100=0.08%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=35.36%, sys=64.61%, ctx=40, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=917226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=512MiB/s (537MB/s), 512MiB/s-512MiB/s (537MB/s-537MB/s), io=3583MiB (3757MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_2_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733892,
  "timestamp_ms" : 1666733892681,
  "time" : "Tue Oct 25 23:38:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 3749478400,
        "io_kbytes" : 3661600,
        "bw_bytes" : 535563262,
        "bw" : 523010,
        "iops" : 130752.606770,
        "runtime" : 7001,
        "total_ios" : 915399,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1552,
          "max" : 175259,
          "mean" : 1626.508294,
          "stddev" : 223.366531,
          "N" : 915399
        },
        "clat_ns" : {
          "min" : 711,
          "max" : 181591,
          "mean" : 12431.741996,
          "stddev" : 3780.590270,
          "N" : 915399,
          "percentile" : {
            "1.000000" : 10560,
            "5.000000" : 10560,
            "10.000000" : 10688,
            "20.000000" : 10816,
            "30.000000" : 10944,
            "40.000000" : 11072,
            "50.000000" : 12352,
            "60.000000" : 12480,
            "70.000000" : 12608,
            "80.000000" : 12736,
            "90.000000" : 12864,
            "95.000000" : 13376,
            "99.000000" : 29824,
            "99.500000" : 37120,
            "99.900000" : 50432,
            "99.950000" : 63744,
            "99.990000" : 80384
          }
        },
        "lat_ns" : {
          "min" : 9377,
          "max" : 188053,
          "mean" : 14058.250290,
          "stddev" : 3793.051475,
          "N" : 915399
        },
        "bw_min" : 521616,
        "bw_max" : 524296,
        "bw_agg" : 100.000000,
        "bw_mean" : 523131.285714,
        "bw_dev" : 699.172965,
        "bw_samples" : 14,
        "iops_min" : 130404,
        "iops_max" : 131074,
        "iops_mean" : 130782.785714,
        "iops_stddev" : 174.475116,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.957143,
      "sys_cpu" : 65.014286,
      "ctx" : 17,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 49.974055,
        "2" : 50.025945,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.063470,
        "20" : 96.510811,
        "50" : 3.320410,
        "100" : 0.101267,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87122: Tue Oct 25 23:38:12 2022
  read: IOPS=131k, BW=511MiB/s (536MB/s)(3576MiB/7001msec)
    slat (nsec): min=1552, max=175259, avg=1626.51, stdev=223.37
    clat (nsec): min=711, max=181591, avg=12431.74, stdev=3780.59
     lat (usec): min=9, max=188, avg=14.06, stdev= 3.79
    clat percentiles (nsec):
     |  1.00th=[10560],  5.00th=[10560], 10.00th=[10688], 20.00th=[10816],
     | 30.00th=[10944], 40.00th=[11072], 50.00th=[12352], 60.00th=[12480],
     | 70.00th=[12608], 80.00th=[12736], 90.00th=[12864], 95.00th=[13376],
     | 99.00th=[29824], 99.50th=[37120], 99.90th=[50432], 99.95th=[63744],
     | 99.99th=[80384]
   bw (  KiB/s): min=521616, max=524296, per=100.00%, avg=523131.29, stdev=699.17, samples=14
   iops        : min=130404, max=131074, avg=130782.79, stdev=174.48, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.06%, 20=96.51%, 50=3.32%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=34.96%, sys=65.01%, ctx=17, majf=0, minf=0
  IO depths    : 1=50.0%, 2=50.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=915399,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=511MiB/s (536MB/s), 511MiB/s-511MiB/s (536MB/s-536MB/s), io=3576MiB (3749MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ rm /tmp/fio-output_4K_2_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733906,
  "timestamp_ms" : 1666733906474,
  "time" : "Tue Oct 25 23:38:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2690433024,
        "io_kbytes" : 2627376,
        "bw_bytes" : 384292675,
        "bw" : 375285,
        "iops" : 93821.311241,
        "runtime" : 7001,
        "total_ios" : 656843,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2504,
          "max" : 178896,
          "mean" : 2877.458288,
          "stddev" : 468.361840,
          "N" : 656843
        },
        "clat_ns" : {
          "min" : 951,
          "max" : 1352827,
          "mean" : 17919.024456,
          "stddev" : 5851.439789,
          "N" : 656844,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 12864,
            "10.000000" : 12992,
            "20.000000" : 12992,
            "30.000000" : 13120,
            "40.000000" : 13120,
            "50.000000" : 21376,
            "60.000000" : 21376,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 21632,
            "95.000000" : 22144,
            "99.000000" : 35072,
            "99.500000" : 40192,
            "99.900000" : 50944,
            "99.950000" : 64256,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 12153,
          "max" : 1355933,
          "mean" : 20796.483029,
          "stddev" : 6068.884642,
          "N" : 656844
        },
        "bw_min" : 374056,
        "bw_max" : 377312,
        "bw_agg" : 100.000000,
        "bw_mean" : 375377.071429,
        "bw_dev" : 779.433371,
        "bw_samples" : 14,
        "iops_min" : 93514,
        "iops_max" : 94328,
        "iops_mean" : 93844.214286,
        "iops_stddev" : 194.825436,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.200000,
      "sys_cpu" : 45.857143,
      "ctx" : 328203,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 47.701962,
        "50" : 52.184160,
        "100" : 0.103678,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87161: Tue Oct 25 23:38:26 2022
  read: IOPS=93.8k, BW=366MiB/s (384MB/s)(2566MiB/7001msec)
    slat (nsec): min=2504, max=178896, avg=2877.46, stdev=468.36
    clat (nsec): min=951, max=1352.8k, avg=17919.02, stdev=5851.44
     lat (usec): min=12, max=1355, avg=20.80, stdev= 6.07
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12864], 10.00th=[12992], 20.00th=[12992],
     | 30.00th=[13120], 40.00th=[13120], 50.00th=[21376], 60.00th=[21376],
     | 70.00th=[21376], 80.00th=[21632], 90.00th=[21632], 95.00th=[22144],
     | 99.00th=[35072], 99.50th=[40192], 99.90th=[50944], 99.95th=[64256],
     | 99.99th=[77312]
   bw (  KiB/s): min=374056, max=377312, per=100.00%, avg=375377.07, stdev=779.43, samples=14
   iops        : min=93514, max=94328, avg=93844.21, stdev=194.83, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=47.70%, 50=52.18%, 100=0.10%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=15.20%, sys=45.86%, ctx=328203, majf=0, minf=1
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=656843,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=366MiB/s (384MB/s), 366MiB/s-366MiB/s (384MB/s-384MB/s), io=2566MiB (2690MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733920,
  "timestamp_ms" : 1666733920115,
  "time" : "Tue Oct 25 23:38:40 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2698158080,
        "io_kbytes" : 2634920,
        "bw_bytes" : 385396097,
        "bw" : 376363,
        "iops" : 94090.701328,
        "runtime" : 7001,
        "total_ios" : 658729,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2534,
          "max" : 47179,
          "mean" : 2900.782150,
          "stddev" : 418.117679,
          "N" : 658729
        },
        "clat_ns" : {
          "min" : 921,
          "max" : 118352,
          "mean" : 17829.021652,
          "stddev" : 5403.400026,
          "N" : 658730,
          "percentile" : {
            "1.000000" : 12608,
            "5.000000" : 12736,
            "10.000000" : 12736,
            "20.000000" : 12864,
            "30.000000" : 12864,
            "40.000000" : 12992,
            "50.000000" : 21376,
            "60.000000" : 21376,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 21632,
            "95.000000" : 22144,
            "99.000000" : 34560,
            "99.500000" : 39680,
            "99.900000" : 48896,
            "99.950000" : 61184,
            "99.990000" : 76288
          }
        },
        "lat_ns" : {
          "min" : 12253,
          "max" : 121407,
          "mean" : 20729.804006,
          "stddev" : 5628.504721,
          "N" : 658730
        },
        "bw_min" : 375336,
        "bw_max" : 377432,
        "bw_agg" : 100.000000,
        "bw_mean" : 376463.000000,
        "bw_dev" : 622.233570,
        "bw_samples" : 14,
        "iops_min" : 93834,
        "iops_max" : 94358,
        "iops_mean" : 94115.714286,
        "iops_stddev" : 155.503860,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.242857,
      "sys_cpu" : 46.571429,
      "ctx" : 329136,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010019,
        "20" : 47.720686,
        "50" : 52.176995,
        "100" : 0.090629,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87201: Tue Oct 25 23:38:40 2022
  read: IOPS=94.1k, BW=368MiB/s (385MB/s)(2573MiB/7001msec)
    slat (nsec): min=2534, max=47179, avg=2900.78, stdev=418.12
    clat (nsec): min=921, max=118352, avg=17829.02, stdev=5403.40
     lat (usec): min=12, max=121, avg=20.73, stdev= 5.63
    clat percentiles (nsec):
     |  1.00th=[12608],  5.00th=[12736], 10.00th=[12736], 20.00th=[12864],
     | 30.00th=[12864], 40.00th=[12992], 50.00th=[21376], 60.00th=[21376],
     | 70.00th=[21376], 80.00th=[21632], 90.00th=[21632], 95.00th=[22144],
     | 99.00th=[34560], 99.50th=[39680], 99.90th=[48896], 99.95th=[61184],
     | 99.99th=[76288]
   bw (  KiB/s): min=375336, max=377432, per=100.00%, avg=376463.00, stdev=622.23, samples=14
   iops        : min=93834, max=94358, avg=94115.71, stdev=155.50, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=47.72%, 50=52.18%
  lat (usec)   : 100=0.09%, 250=0.01%
  cpu          : usr=15.24%, sys=46.57%, ctx=329136, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=658729,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=368MiB/s (385MB/s), 368MiB/s-368MiB/s (385MB/s-385MB/s), io=2573MiB (2698MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_2_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_2_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="2" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_2_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_2_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=2
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733933,
  "timestamp_ms" : 1666733933796,
  "time" : "Tue Oct 25 23:38:53 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "2",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 2710253568,
        "io_kbytes" : 2646732,
        "bw_bytes" : 387123777,
        "bw" : 378050,
        "iops" : 94512.498215,
        "runtime" : 7001,
        "total_ios" : 661682,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2494,
          "max" : 179777,
          "mean" : 2858.011643,
          "stddev" : 548.309512,
          "N" : 661682
        },
        "clat_ns" : {
          "min" : 902,
          "max" : 195327,
          "mean" : 17806.183939,
          "stddev" : 5345.044073,
          "N" : 661683,
          "percentile" : {
            "1.000000" : 12736,
            "5.000000" : 12864,
            "10.000000" : 12864,
            "20.000000" : 12992,
            "30.000000" : 12992,
            "40.000000" : 13120,
            "50.000000" : 21120,
            "60.000000" : 21376,
            "70.000000" : 21376,
            "80.000000" : 21376,
            "90.000000" : 21376,
            "95.000000" : 21888,
            "99.000000" : 35072,
            "99.500000" : 40192,
            "99.900000" : 50944,
            "99.950000" : 64768,
            "99.990000" : 77312
          }
        },
        "lat_ns" : {
          "min" : 12413,
          "max" : 201298,
          "mean" : 20664.195956,
          "stddev" : 5573.558208,
          "N" : 661683
        },
        "bw_min" : 377240,
        "bw_max" : 379552,
        "bw_agg" : 100.000000,
        "bw_mean" : 378145.928571,
        "bw_dev" : 652.061638,
        "bw_samples" : 14,
        "iops_min" : 94310,
        "iops_max" : 94888,
        "iops_mean" : 94536.428571,
        "iops_stddev" : 163.055778,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.442857,
      "sys_cpu" : 43.571429,
      "ctx" : 330603,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 100.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 47.726551,
        "50" : 52.157683,
        "100" : 0.103373,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 2,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=87241: Tue Oct 25 23:38:53 2022
  read: IOPS=94.5k, BW=369MiB/s (387MB/s)(2585MiB/7001msec)
    slat (nsec): min=2494, max=179777, avg=2858.01, stdev=548.31
    clat (nsec): min=902, max=195327, avg=17806.18, stdev=5345.04
     lat (usec): min=12, max=201, avg=20.66, stdev= 5.57
    clat percentiles (nsec):
     |  1.00th=[12736],  5.00th=[12864], 10.00th=[12864], 20.00th=[12992],
     | 30.00th=[12992], 40.00th=[13120], 50.00th=[21120], 60.00th=[21376],
     | 70.00th=[21376], 80.00th=[21376], 90.00th=[21376], 95.00th=[21888],
     | 99.00th=[35072], 99.50th=[40192], 99.90th=[50944], 99.95th=[64768],
     | 99.99th=[77312]
   bw (  KiB/s): min=377240, max=379552, per=100.00%, avg=378145.93, stdev=652.06, samples=14
   iops        : min=94310, max=94888, avg=94536.43, stdev=163.06, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=47.73%, 50=52.16%, 100=0.10%
  lat (usec)   : 250=0.01%
  cpu          : usr=17.44%, sys=43.57%, ctx=330603, majf=0, minf=0
  IO depths    : 1=0.0%, 2=100.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=661682,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=2

Run status group 0 (all jobs):
   READ: bw=369MiB/s (387MB/s), 369MiB/s-369MiB/s (387MB/s-387MB/s), io=2585MiB (2710MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733947,
  "timestamp_ms" : 1666733947431,
  "time" : "Tue Oct 25 23:39:07 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4239355904,
        "io_kbytes" : 4139996,
        "bw_bytes" : 605535766,
        "bw" : 591343,
        "iops" : 147835.309242,
        "runtime" : 7001,
        "total_ios" : 1034995,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1863,
          "max" : 68268,
          "mean" : 2134.228236,
          "stddev" : 423.418030,
          "N" : 1034996
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 170530,
          "mean" : 24422.255110,
          "stddev" : 7557.409280,
          "N" : 1034998,
          "percentile" : {
            "1.000000" : 14528,
            "5.000000" : 17024,
            "10.000000" : 17024,
            "20.000000" : 17792,
            "30.000000" : 18048,
            "40.000000" : 18560,
            "50.000000" : 28032,
            "60.000000" : 28032,
            "70.000000" : 28288,
            "80.000000" : 28800,
            "90.000000" : 29056,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 53504,
            "99.900000" : 82432,
            "99.950000" : 97792,
            "99.990000" : 112128
          }
        },
        "lat_ns" : {
          "min" : 13415,
          "max" : 172494,
          "mean" : 26556.483509,
          "stddev" : 7660.333802,
          "N" : 1034998
        },
        "bw_min" : 590184,
        "bw_max" : 592416,
        "bw_agg" : 100.000000,
        "bw_mean" : 591485.071429,
        "bw_dev" : 642.692948,
        "bw_samples" : 14,
        "iops_min" : 147546,
        "iops_max" : 148104,
        "iops_mean" : 147871.214286,
        "iops_stddev" : 160.657389,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.742857,
      "sys_cpu" : 57.785714,
      "ctx" : 266084,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 44.522920,
        "50" : 53.856106,
        "100" : 1.573244,
        "250" : 0.045604,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87281: Tue Oct 25 23:39:07 2022
  read: IOPS=148k, BW=577MiB/s (606MB/s)(4043MiB/7001msec)
    slat (nsec): min=1863, max=68268, avg=2134.23, stdev=423.42
    clat (nsec): min=210, max=170530, avg=24422.26, stdev=7557.41
     lat (usec): min=13, max=172, avg=26.56, stdev= 7.66
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   18], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   19], 40.00th=[   19], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   29], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   54], 99.90th=[   83], 99.95th=[   98],
     | 99.99th=[  113]
   bw (  KiB/s): min=590184, max=592416, per=100.00%, avg=591485.07, stdev=642.69, samples=14
   iops        : min=147546, max=148104, avg=147871.21, stdev=160.66, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=44.52%, 50=53.86%, 100=1.57%
  lat (usec)   : 250=0.05%
  cpu          : usr=14.74%, sys=57.79%, ctx=266084, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1034995,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=577MiB/s (606MB/s), 577MiB/s-577MiB/s (606MB/s-606MB/s), io=4043MiB (4239MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733961,
  "timestamp_ms" : 1666733961073,
  "time" : "Tue Oct 25 23:39:21 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4248465408,
        "io_kbytes" : 4148892,
        "bw_bytes" : 606836938,
        "bw" : 592614,
        "iops" : 148152.978146,
        "runtime" : 7001,
        "total_ios" : 1037219,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1874,
          "max" : 27602,
          "mean" : 2143.573774,
          "stddev" : 413.583393,
          "N" : 1037220
        },
        "clat_ns" : {
          "min" : 2985,
          "max" : 204163,
          "mean" : 24353.154686,
          "stddev" : 7561.186398,
          "N" : 1037222,
          "percentile" : {
            "1.000000" : 16768,
            "5.000000" : 17024,
            "10.000000" : 17024,
            "20.000000" : 17792,
            "30.000000" : 17792,
            "40.000000" : 18560,
            "50.000000" : 28032,
            "60.000000" : 28032,
            "70.000000" : 28288,
            "80.000000" : 28800,
            "90.000000" : 29056,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 53504,
            "99.900000" : 81408,
            "99.950000" : 96768,
            "99.990000" : 113152
          }
        },
        "lat_ns" : {
          "min" : 13285,
          "max" : 206557,
          "mean" : 26496.728306,
          "stddev" : 7666.456687,
          "N" : 1037222
        },
        "bw_min" : 591632,
        "bw_max" : 593816,
        "bw_agg" : 100.000000,
        "bw_mean" : 592752.785714,
        "bw_dev" : 705.015022,
        "bw_samples" : 14,
        "iops_min" : 147908,
        "iops_max" : 148454,
        "iops_mean" : 148188.142857,
        "iops_stddev" : 176.173803,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.328571,
      "sys_cpu" : 55.942857,
      "ctx" : 266741,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 45.803249,
        "50" : 52.586484,
        "100" : 1.563893,
        "250" : 0.044157,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87319: Tue Oct 25 23:39:21 2022
  read: IOPS=148k, BW=579MiB/s (607MB/s)(4052MiB/7001msec)
    slat (nsec): min=1874, max=27602, avg=2143.57, stdev=413.58
    clat (usec): min=2, max=204, avg=24.35, stdev= 7.56
     lat (usec): min=13, max=206, avg=26.50, stdev= 7.67
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   18], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   29], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   54], 99.90th=[   82], 99.95th=[   97],
     | 99.99th=[  114]
   bw (  KiB/s): min=591632, max=593816, per=100.00%, avg=592752.79, stdev=705.02, samples=14
   iops        : min=147908, max=148454, avg=148188.14, stdev=176.17, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=45.80%, 50=52.59%, 100=1.56%
  lat (usec)   : 250=0.04%
  cpu          : usr=16.33%, sys=55.94%, ctx=266741, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1037219,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=579MiB/s (607MB/s), 579MiB/s-579MiB/s (607MB/s-607MB/s), io=4052MiB (4248MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733974,
  "timestamp_ms" : 1666733974770,
  "time" : "Tue Oct 25 23:39:34 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4263362560,
        "io_kbytes" : 4163440,
        "bw_bytes" : 608964799,
        "bw" : 594692,
        "iops" : 148672.618197,
        "runtime" : 7001,
        "total_ios" : 1040857,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1883,
          "max" : 30347,
          "mean" : 2119.325331,
          "stddev" : 420.060474,
          "N" : 1040857
        },
        "clat_ns" : {
          "min" : 251,
          "max" : 149521,
          "mean" : 24289.765743,
          "stddev" : 7494.997171,
          "N" : 1040860,
          "percentile" : {
            "1.000000" : 15168,
            "5.000000" : 17024,
            "10.000000" : 17024,
            "20.000000" : 17792,
            "30.000000" : 17792,
            "40.000000" : 18560,
            "50.000000" : 27776,
            "60.000000" : 28032,
            "70.000000" : 28288,
            "80.000000" : 28800,
            "90.000000" : 29056,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 53504,
            "99.900000" : 81408,
            "99.950000" : 97792,
            "99.990000" : 113152
          }
        },
        "lat_ns" : {
          "min" : 14096,
          "max" : 151475,
          "mean" : 26409.091097,
          "stddev" : 7588.653299,
          "N" : 1040860
        },
        "bw_min" : 593880,
        "bw_max" : 595576,
        "bw_agg" : 100.000000,
        "bw_mean" : 594850.642857,
        "bw_dev" : 491.431129,
        "bw_samples" : 14,
        "iops_min" : 148470,
        "iops_max" : 148894,
        "iops_mean" : 148712.642857,
        "iops_stddev" : 122.870543,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.671429,
      "sys_cpu" : 58.242857,
      "ctx" : 267603,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 45.125891,
        "50" : 53.340180,
        "100" : 1.487716,
        "250" : 0.043714,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=87357: Tue Oct 25 23:39:34 2022
  read: IOPS=149k, BW=581MiB/s (609MB/s)(4066MiB/7001msec)
    slat (nsec): min=1883, max=30347, avg=2119.33, stdev=420.06
    clat (nsec): min=251, max=149521, avg=24289.77, stdev=7495.00
     lat (usec): min=14, max=151, avg=26.41, stdev= 7.59
    clat percentiles (usec):
     |  1.00th=[   16],  5.00th=[   18], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   28], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   29], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   54], 99.90th=[   82], 99.95th=[   98],
     | 99.99th=[  114]
   bw (  KiB/s): min=593880, max=595576, per=100.00%, avg=594850.64, stdev=491.43, samples=14
   iops        : min=148470, max=148894, avg=148712.64, stdev=122.87, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=45.13%, 50=53.34%, 100=1.49%
  lat (usec)   : 250=0.04%
  cpu          : usr=13.67%, sys=58.24%, ctx=267603, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1040857,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=581MiB/s (609MB/s), 581MiB/s-581MiB/s (609MB/s-609MB/s), io=4066MiB (4263MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733085,
  "timestamp_ms" : 1666733085963,
  "time" : "Tue Oct 25 23:24:45 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2405982208,
        "io_kbytes" : 2349592,
        "bw_bytes" : 343662649,
        "bw" : 335608,
        "iops" : 83902.013998,
        "runtime" : 7001,
        "total_ios" : 587398,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 4368,
          "mean" : 924.473922,
          "stddev" : 69.051652,
          "N" : 587398
        },
        "clat_ns" : {
          "min" : 9077,
          "max" : 113724,
          "mean" : 10706.197505,
          "stddev" : 2217.342043,
          "N" : 587398,
          "percentile" : {
            "1.000000" : 9920,
            "5.000000" : 10048,
            "10.000000" : 10048,
            "20.000000" : 10176,
            "30.000000" : 10304,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10432,
            "70.000000" : 10560,
            "80.000000" : 10560,
            "90.000000" : 10688,
            "95.000000" : 11072,
            "99.000000" : 20096,
            "99.500000" : 27520,
            "99.900000" : 30080,
            "99.950000" : 48384,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 10510,
          "max" : 114625,
          "mean" : 11630.671427,
          "stddev" : 2218.545088,
          "N" : 587398
        },
        "bw_min" : 335280,
        "bw_max" : 336168,
        "bw_agg" : 100.000000,
        "bw_mean" : 335695.428571,
        "bw_dev" : 291.592369,
        "bw_samples" : 14,
        "iops_min" : 83820,
        "iops_max" : 84042,
        "iops_mean" : 83923.857143,
        "iops_stddev" : 72.898092,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 3.081897,
        "20" : 95.920653,
        "50" : 0.950122,
        "100" : 0.047157,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84633: Tue Oct 25 23:24:45 2022
  read: IOPS=83.9k, BW=328MiB/s (344MB/s)(2295MiB/7001msec)
    slat (nsec): min=711, max=4368, avg=924.47, stdev=69.05
    clat (usec): min=9, max=113, avg=10.71, stdev= 2.22
     lat (usec): min=10, max=114, avg=11.63, stdev= 2.22
    clat percentiles (nsec):
     |  1.00th=[ 9920],  5.00th=[10048], 10.00th=[10048], 20.00th=[10176],
     | 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
     | 70.00th=[10560], 80.00th=[10560], 90.00th=[10688], 95.00th=[11072],
     | 99.00th=[20096], 99.50th=[27520], 99.90th=[30080], 99.95th=[48384],
     | 99.99th=[68096]
   bw (  KiB/s): min=335280, max=336168, per=100.00%, avg=335695.43, stdev=291.59, samples=14
   iops        : min=83820, max=84042, avg=83923.86, stdev=72.90, samples=14
  lat (usec)   : 10=3.08%, 20=95.92%, 50=0.95%, 100=0.05%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=16, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=587398,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2295MiB (2406MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733988,
  "timestamp_ms" : 1666733988393,
  "time" : "Tue Oct 25 23:39:48 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4244062208,
        "io_kbytes" : 4144592,
        "bw_bytes" : 606208000,
        "bw" : 592000,
        "iops" : 147999.428653,
        "runtime" : 7001,
        "total_ios" : 1036144,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1973,
          "max" : 32401,
          "mean" : 2217.550113,
          "stddev" : 422.893349,
          "N" : 1036145
        },
        "clat_ns" : {
          "min" : 260,
          "max" : 149982,
          "mean" : 24348.226217,
          "stddev" : 7711.887494,
          "N" : 1036147,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 17536,
            "30.000000" : 17536,
            "40.000000" : 18304,
            "50.000000" : 28288,
            "60.000000" : 28288,
            "70.000000" : 28544,
            "80.000000" : 29056,
            "90.000000" : 29312,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 53504,
            "99.900000" : 80384,
            "99.950000" : 96768,
            "99.990000" : 113152
          }
        },
        "lat_ns" : {
          "min" : 13815,
          "max" : 152456,
          "mean" : 26565.776468,
          "stddev" : 7822.726275,
          "N" : 1036147
        },
        "bw_min" : 590720,
        "bw_max" : 593480,
        "bw_agg" : 100.000000,
        "bw_mean" : 592140.500000,
        "bw_dev" : 670.367379,
        "bw_samples" : 14,
        "iops_min" : 147680,
        "iops_max" : 148370,
        "iops_mean" : 148035.071429,
        "iops_stddev" : 167.612403,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.842857,
      "sys_cpu" : 56.142857,
      "ctx" : 265858,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 45.727042,
        "50" : 52.683121,
        "100" : 1.546310,
        "250" : 0.040824,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87395: Tue Oct 25 23:39:48 2022
  read: IOPS=148k, BW=578MiB/s (606MB/s)(4047MiB/7001msec)
    slat (nsec): min=1973, max=32401, avg=2217.55, stdev=422.89
    clat (nsec): min=260, max=149982, avg=24348.23, stdev=7711.89
     lat (usec): min=13, max=152, avg=26.57, stdev= 7.82
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   30], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   54], 99.90th=[   81], 99.95th=[   97],
     | 99.99th=[  114]
   bw (  KiB/s): min=590720, max=593480, per=100.00%, avg=592140.50, stdev=670.37, samples=14
   iops        : min=147680, max=148370, avg=148035.07, stdev=167.61, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=45.73%, 50=52.68%, 100=1.55%
  lat (usec)   : 250=0.04%
  cpu          : usr=17.84%, sys=56.14%, ctx=265858, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1036144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=578MiB/s (606MB/s), 578MiB/s-578MiB/s (606MB/s-606MB/s), io=4047MiB (4244MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734002,
  "timestamp_ms" : 1666734002404,
  "time" : "Tue Oct 25 23:40:02 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4233482240,
        "io_kbytes" : 4134260,
        "bw_bytes" : 604696791,
        "bw" : 590524,
        "iops" : 147630.624197,
        "runtime" : 7001,
        "total_ios" : 1033562,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1973,
          "max" : 28543,
          "mean" : 2227.234882,
          "stddev" : 433.965898,
          "N" : 1033562
        },
        "clat_ns" : {
          "min" : 291,
          "max" : 150883,
          "mean" : 24397.982275,
          "stddev" : 7660.972958,
          "N" : 1033565,
          "percentile" : {
            "1.000000" : 16512,
            "5.000000" : 16768,
            "10.000000" : 16768,
            "20.000000" : 17536,
            "30.000000" : 17792,
            "40.000000" : 18304,
            "50.000000" : 28288,
            "60.000000" : 28544,
            "70.000000" : 28544,
            "80.000000" : 29056,
            "90.000000" : 29312,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 52992,
            "99.900000" : 76288,
            "99.950000" : 94720,
            "99.990000" : 112128
          }
        },
        "lat_ns" : {
          "min" : 13616,
          "max" : 153037,
          "mean" : 26625.217129,
          "stddev" : 7783.374398,
          "N" : 1033565
        },
        "bw_min" : 589488,
        "bw_max" : 591486,
        "bw_agg" : 100.000000,
        "bw_mean" : 590663.285714,
        "bw_dev" : 586.088787,
        "bw_samples" : 14,
        "iops_min" : 147372,
        "iops_max" : 147871,
        "iops_mean" : 147665.785714,
        "iops_stddev" : 146.583753,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.671429,
      "sys_cpu" : 55.157143,
      "ctx" : 265038,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 45.679117,
        "50" : 52.747102,
        "100" : 1.533628,
        "250" : 0.037540,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87435: Tue Oct 25 23:40:02 2022
  read: IOPS=148k, BW=577MiB/s (605MB/s)(4037MiB/7001msec)
    slat (nsec): min=1973, max=28543, avg=2227.23, stdev=433.97
    clat (nsec): min=291, max=150883, avg=24397.98, stdev=7660.97
     lat (usec): min=13, max=153, avg=26.63, stdev= 7.78
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   30], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   53], 99.90th=[   77], 99.95th=[   95],
     | 99.99th=[  113]
   bw (  KiB/s): min=589488, max=591486, per=100.00%, avg=590663.29, stdev=586.09, samples=14
   iops        : min=147372, max=147871, avg=147665.79, stdev=146.58, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=45.68%, 50=52.75%, 100=1.53%
  lat (usec)   : 250=0.04%
  cpu          : usr=18.67%, sys=55.16%, ctx=265038, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1033562,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=577MiB/s (605MB/s), 577MiB/s-577MiB/s (605MB/s-605MB/s), io=4037MiB (4233MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734016,
  "timestamp_ms" : 1666734016263,
  "time" : "Tue Oct 25 23:40:16 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 4226142208,
        "io_kbytes" : 4127092,
        "bw_bytes" : 603648365,
        "bw" : 589500,
        "iops" : 147374.517926,
        "runtime" : 7001,
        "total_ios" : 1031769,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1993,
          "max" : 26640,
          "mean" : 2267.380921,
          "stddev" : 434.200456,
          "N" : 1031770
        },
        "clat_ns" : {
          "min" : 310,
          "max" : 200466,
          "mean" : 24390.233048,
          "stddev" : 7872.725576,
          "N" : 1031772,
          "percentile" : {
            "1.000000" : 16192,
            "5.000000" : 16512,
            "10.000000" : 16768,
            "20.000000" : 17280,
            "30.000000" : 17536,
            "40.000000" : 18304,
            "50.000000" : 28288,
            "60.000000" : 28544,
            "70.000000" : 28800,
            "80.000000" : 29056,
            "90.000000" : 29568,
            "95.000000" : 34048,
            "99.000000" : 51968,
            "99.500000" : 53504,
            "99.900000" : 84480,
            "99.950000" : 99840,
            "99.990000" : 115200
          }
        },
        "lat_ns" : {
          "min" : 13556,
          "max" : 203111,
          "mean" : 26657.614235,
          "stddev" : 7990.434757,
          "N" : 1031772
        },
        "bw_min" : 587911,
        "bw_max" : 591096,
        "bw_agg" : 100.000000,
        "bw_mean" : 589642.214286,
        "bw_dev" : 928.622982,
        "bw_samples" : 14,
        "iops_min" : 146977,
        "iops_max" : 147774,
        "iops_mean" : 147410.500000,
        "iops_stddev" : 232.283232,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.514286,
      "sys_cpu" : 56.057143,
      "ctx" : 264480,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 45.311111,
        "50" : 53.090178,
        "100" : 1.546858,
        "250" : 0.049139,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=87473: Tue Oct 25 23:40:16 2022
  read: IOPS=147k, BW=576MiB/s (604MB/s)(4030MiB/7001msec)
    slat (nsec): min=1993, max=26640, avg=2267.38, stdev=434.20
    clat (nsec): min=310, max=200466, avg=24390.23, stdev=7872.73
     lat (usec): min=13, max=203, avg=26.66, stdev= 7.99
    clat percentiles (usec):
     |  1.00th=[   17],  5.00th=[   17], 10.00th=[   17], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   19], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   30], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   52], 99.50th=[   54], 99.90th=[   85], 99.95th=[  100],
     | 99.99th=[  116]
   bw (  KiB/s): min=587911, max=591096, per=100.00%, avg=589642.21, stdev=928.62, samples=14
   iops        : min=146977, max=147774, avg=147410.50, stdev=232.28, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=45.31%, 50=53.09%, 100=1.55%
  lat (usec)   : 250=0.05%
  cpu          : usr=18.51%, sys=56.06%, ctx=264480, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1031769,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=576MiB/s (604MB/s), 576MiB/s-576MiB/s (604MB/s-604MB/s), io=4030MiB (4226MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734031,
  "timestamp_ms" : 1666734031180,
  "time" : "Tue Oct 25 23:40:31 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 5484310528,
        "io_kbytes" : 5355772,
        "bw_bytes" : 783361023,
        "bw" : 765000,
        "iops" : 191249.821454,
        "runtime" : 7001,
        "total_ios" : 1338940,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 661,
          "max" : 38161,
          "mean" : 943.770155,
          "stddev" : 157.273061,
          "N" : 1338940
        },
        "clat_ns" : {
          "min" : 1282,
          "max" : 194796,
          "mean" : 19434.049774,
          "stddev" : 6553.654058,
          "N" : 1338942,
          "percentile" : {
            "1.000000" : 11200,
            "5.000000" : 12480,
            "10.000000" : 12992,
            "20.000000" : 15936,
            "30.000000" : 16320,
            "40.000000" : 17024,
            "50.000000" : 18048,
            "60.000000" : 19328,
            "70.000000" : 20352,
            "80.000000" : 22656,
            "90.000000" : 24448,
            "95.000000" : 32640,
            "99.000000" : 44288,
            "99.500000" : 47360,
            "99.900000" : 77312,
            "99.950000" : 92672,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 11071,
          "max" : 195707,
          "mean" : 20377.819873,
          "stddev" : 6563.731917,
          "N" : 1338942
        },
        "bw_min" : 764144,
        "bw_max" : 766368,
        "bw_agg" : 100.000000,
        "bw_mean" : 765234.071429,
        "bw_dev" : 634.098083,
        "bw_samples" : 14,
        "iops_min" : 191036,
        "iops_max" : 191592,
        "iops_mean" : 191308.357143,
        "iops_stddev" : 158.356757,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 9,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 19.636429,
        "4" : 80.350725,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 65.775315,
        "50" : 33.972023,
        "100" : 0.223236,
        "250" : 0.027634,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87522: Tue Oct 25 23:40:31 2022
  read: IOPS=191k, BW=747MiB/s (783MB/s)(5230MiB/7001msec)
    slat (nsec): min=661, max=38161, avg=943.77, stdev=157.27
    clat (nsec): min=1282, max=194796, avg=19434.05, stdev=6553.65
     lat (usec): min=11, max=195, avg=20.38, stdev= 6.56
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   13], 20.00th=[   16],
     | 30.00th=[   17], 40.00th=[   18], 50.00th=[   19], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   25], 95.00th=[   33],
     | 99.00th=[   45], 99.50th=[   48], 99.90th=[   78], 99.95th=[   93],
     | 99.99th=[  106]
   bw (  KiB/s): min=764144, max=766368, per=100.00%, avg=765234.07, stdev=634.10, samples=14
   iops        : min=191036, max=191592, avg=191308.36, stdev=158.36, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=65.78%, 50=33.97%
  lat (usec)   : 100=0.22%, 250=0.03%
  cpu          : usr=99.99%, sys=0.00%, ctx=9, majf=0, minf=0
  IO depths    : 1=0.1%, 2=19.6%, 4=80.4%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1338940,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=747MiB/s (783MB/s), 747MiB/s-747MiB/s (783MB/s-783MB/s), io=5230MiB (5484MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734046,
  "timestamp_ms" : 1666734046275,
  "time" : "Tue Oct 25 23:40:46 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 5460344832,
        "io_kbytes" : 5332368,
        "bw_bytes" : 779937842,
        "bw" : 761658,
        "iops" : 190414.083702,
        "runtime" : 7001,
        "total_ios" : 1333089,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 651,
          "max" : 38282,
          "mean" : 939.539594,
          "stddev" : 143.340985,
          "N" : 1333089
        },
        "clat_ns" : {
          "min" : 1283,
          "max" : 151294,
          "mean" : 19526.900070,
          "stddev" : 6576.455932,
          "N" : 1333092,
          "percentile" : {
            "1.000000" : 11328,
            "5.000000" : 12480,
            "10.000000" : 13120,
            "20.000000" : 16064,
            "30.000000" : 16512,
            "40.000000" : 17024,
            "50.000000" : 18048,
            "60.000000" : 19328,
            "70.000000" : 20352,
            "80.000000" : 22912,
            "90.000000" : 24448,
            "95.000000" : 32640,
            "99.000000" : 44288,
            "99.500000" : 47872,
            "99.900000" : 79360,
            "99.950000" : 93696,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 10961,
          "max" : 152466,
          "mean" : 20466.439579,
          "stddev" : 6586.533800,
          "N" : 1333092
        },
        "bw_min" : 760152,
        "bw_max" : 764288,
        "bw_agg" : 100.000000,
        "bw_mean" : 761844.357143,
        "bw_dev" : 972.547456,
        "bw_samples" : 14,
        "iops_min" : 190038,
        "iops_max" : 191072,
        "iops_mean" : 190461.071429,
        "iops_stddev" : 243.120883,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 19,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 19.228874,
        "4" : 80.757024,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 65.134736,
        "50" : 34.590939,
        "100" : 0.244395,
        "250" : 0.028730,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87573: Tue Oct 25 23:40:46 2022
  read: IOPS=190k, BW=744MiB/s (780MB/s)(5207MiB/7001msec)
    slat (nsec): min=651, max=38282, avg=939.54, stdev=143.34
    clat (nsec): min=1283, max=151294, avg=19526.90, stdev=6576.46
     lat (usec): min=10, max=152, avg=20.47, stdev= 6.59
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   14], 20.00th=[   17],
     | 30.00th=[   17], 40.00th=[   18], 50.00th=[   19], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   25], 95.00th=[   33],
     | 99.00th=[   45], 99.50th=[   48], 99.90th=[   80], 99.95th=[   94],
     | 99.99th=[  106]
   bw (  KiB/s): min=760152, max=764288, per=100.00%, avg=761844.36, stdev=972.55, samples=14
   iops        : min=190038, max=191072, avg=190461.07, stdev=243.12, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=65.13%, 50=34.59%
  lat (usec)   : 100=0.24%, 250=0.03%
  cpu          : usr=99.99%, sys=0.00%, ctx=19, majf=0, minf=0
  IO depths    : 1=0.1%, 2=19.2%, 4=80.8%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1333089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=744MiB/s (780MB/s), 744MiB/s-744MiB/s (780MB/s-780MB/s), io=5207MiB (5460MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734061,
  "timestamp_ms" : 1666734061334,
  "time" : "Tue Oct 25 23:41:01 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 5464141824,
        "io_kbytes" : 5336076,
        "bw_bytes" : 780480191,
        "bw" : 762187,
        "iops" : 190546.493358,
        "runtime" : 7001,
        "total_ios" : 1334016,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 661,
          "max" : 14577,
          "mean" : 940.799248,
          "stddev" : 97.471830,
          "N" : 1334016
        },
        "clat_ns" : {
          "min" : 9919,
          "max" : 193162,
          "mean" : 19508.399007,
          "stddev" : 6583.473141,
          "N" : 1334019,
          "percentile" : {
            "1.000000" : 11328,
            "5.000000" : 12480,
            "10.000000" : 13120,
            "20.000000" : 15936,
            "30.000000" : 16512,
            "40.000000" : 17024,
            "50.000000" : 18048,
            "60.000000" : 19328,
            "70.000000" : 20352,
            "80.000000" : 22912,
            "90.000000" : 24448,
            "95.000000" : 32640,
            "99.000000" : 44288,
            "99.500000" : 47872,
            "99.900000" : 79360,
            "99.950000" : 92672,
            "99.990000" : 107008
          }
        },
        "lat_ns" : {
          "min" : 10850,
          "max" : 193833,
          "mean" : 20449.198191,
          "stddev" : 6592.555887,
          "N" : 1334019
        },
        "bw_min" : 760784,
        "bw_max" : 763832,
        "bw_agg" : 100.000000,
        "bw_mean" : 762373.571429,
        "bw_dev" : 891.857932,
        "bw_samples" : 14,
        "iops_min" : 190196,
        "iops_max" : 190958,
        "iops_mean" : 190593.357143,
        "iops_stddev" : 222.929312,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 19.315660,
        "4" : 80.683590,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 65.097870,
        "50" : 34.639989,
        "100" : 0.232681,
        "250" : 0.029610,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87623: Tue Oct 25 23:41:01 2022
  read: IOPS=191k, BW=744MiB/s (780MB/s)(5211MiB/7001msec)
    slat (nsec): min=661, max=14577, avg=940.80, stdev=97.47
    clat (usec): min=9, max=193, avg=19.51, stdev= 6.58
     lat (usec): min=10, max=193, avg=20.45, stdev= 6.59
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   13], 10.00th=[   14], 20.00th=[   16],
     | 30.00th=[   17], 40.00th=[   18], 50.00th=[   19], 60.00th=[   20],
     | 70.00th=[   21], 80.00th=[   23], 90.00th=[   25], 95.00th=[   33],
     | 99.00th=[   45], 99.50th=[   48], 99.90th=[   80], 99.95th=[   93],
     | 99.99th=[  108]
   bw (  KiB/s): min=760784, max=763832, per=100.00%, avg=762373.57, stdev=891.86, samples=14
   iops        : min=190196, max=190958, avg=190593.36, stdev=222.93, samples=14
  lat (usec)   : 10=0.01%, 20=65.10%, 50=34.64%, 100=0.23%, 250=0.03%
  cpu          : usr=99.99%, sys=0.00%, ctx=18, majf=0, minf=0
  IO depths    : 1=0.1%, 2=19.3%, 4=80.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1334016,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=744MiB/s (780MB/s), 744MiB/s-744MiB/s (780MB/s-780MB/s), io=5211MiB (5464MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734075,
  "timestamp_ms" : 1666734075058,
  "time" : "Tue Oct 25 23:41:15 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4150661120,
        "io_kbytes" : 4053380,
        "bw_bytes" : 592866893,
        "bw" : 578971,
        "iops" : 144742.465362,
        "runtime" : 7001,
        "total_ios" : 1013342,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2234,
          "max" : 35627,
          "mean" : 2427.005221,
          "stddev" : 421.685182,
          "N" : 1013342
        },
        "clat_ns" : {
          "min" : 160,
          "max" : 159249,
          "mean" : 24692.870506,
          "stddev" : 8160.319348,
          "N" : 1013345,
          "percentile" : {
            "1.000000" : 14400,
            "5.000000" : 14528,
            "10.000000" : 14912,
            "20.000000" : 16768,
            "30.000000" : 17536,
            "40.000000" : 20864,
            "50.000000" : 28800,
            "60.000000" : 29056,
            "70.000000" : 29312,
            "80.000000" : 29824,
            "90.000000" : 30080,
            "95.000000" : 34560,
            "99.000000" : 52992,
            "99.500000" : 55552,
            "99.900000" : 80384,
            "99.950000" : 96768,
            "99.990000" : 113152
          }
        },
        "lat_ns" : {
          "min" : 13666,
          "max" : 161533,
          "mean" : 27119.875680,
          "stddev" : 8241.903875,
          "N" : 1013345
        },
        "bw_min" : 577330,
        "bw_max" : 581224,
        "bw_agg" : 100.000000,
        "bw_mean" : 579190.428571,
        "bw_dev" : 1221.195424,
        "bw_samples" : 14,
        "iops_min" : 144332,
        "iops_max" : 145306,
        "iops_mean" : 144797.571429,
        "iops_stddev" : 305.357474,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.071429,
      "sys_cpu" : 60.985714,
      "ctx" : 259323,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 35.179239,
        "50" : 63.216762,
        "100" : 1.556039,
        "250" : 0.043223,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1452241,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 21892,
      "write_ticks" : 0,
      "in_queue" : 21892,
      "util" : 99.049128
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87662: Tue Oct 25 23:41:15 2022
  read: IOPS=145k, BW=565MiB/s (593MB/s)(3958MiB/7001msec)
    slat (nsec): min=2234, max=35627, avg=2427.01, stdev=421.69
    clat (nsec): min=160, max=159249, avg=24692.87, stdev=8160.32
     lat (usec): min=13, max=161, avg=27.12, stdev= 8.24
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   21], 50.00th=[   29], 60.00th=[   30],
     | 70.00th=[   30], 80.00th=[   30], 90.00th=[   31], 95.00th=[   35],
     | 99.00th=[   53], 99.50th=[   56], 99.90th=[   81], 99.95th=[   97],
     | 99.99th=[  114]
   bw (  KiB/s): min=577330, max=581224, per=100.00%, avg=579190.43, stdev=1221.20, samples=14
   iops        : min=144332, max=145306, avg=144797.57, stdev=305.36, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=35.18%, 50=63.22%, 100=1.56%
  lat (usec)   : 250=0.04%
  cpu          : usr=15.07%, sys=60.99%, ctx=259323, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1013342,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=565MiB/s (593MB/s), 565MiB/s-565MiB/s (593MB/s-593MB/s), io=3958MiB (4151MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1452241/0, merge=0/0, ticks=21892/0, in_queue=21892, util=99.05%

$ rm /tmp/fio-output_4K_4_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734088,
  "timestamp_ms" : 1666734088701,
  "time" : "Tue Oct 25 23:41:28 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4203417600,
        "io_kbytes" : 4104900,
        "bw_bytes" : 600402456,
        "bw" : 586330,
        "iops" : 146582.202542,
        "runtime" : 7001,
        "total_ios" : 1026222,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2174,
          "max" : 178164,
          "mean" : 2386.173347,
          "stddev" : 455.786112,
          "N" : 1026222
        },
        "clat_ns" : {
          "min" : 191,
          "max" : 208732,
          "mean" : 24370.688623,
          "stddev" : 8077.609207,
          "N" : 1026225,
          "percentile" : {
            "1.000000" : 14272,
            "5.000000" : 15040,
            "10.000000" : 16512,
            "20.000000" : 16768,
            "30.000000" : 17280,
            "40.000000" : 17792,
            "50.000000" : 28544,
            "60.000000" : 28800,
            "70.000000" : 28800,
            "80.000000" : 29056,
            "90.000000" : 29568,
            "95.000000" : 34560,
            "99.000000" : 52480,
            "99.500000" : 54016,
            "99.900000" : 83456,
            "99.950000" : 99840,
            "99.990000" : 115200
          }
        },
        "lat_ns" : {
          "min" : 11301,
          "max" : 217257,
          "mean" : 26756.861984,
          "stddev" : 8171.492717,
          "N" : 1026225
        },
        "bw_min" : 584704,
        "bw_max" : 588312,
        "bw_agg" : 100.000000,
        "bw_mean" : 586556.714286,
        "bw_dev" : 891.194395,
        "bw_samples" : 14,
        "iops_min" : 146176,
        "iops_max" : 147078,
        "iops_mean" : 146639.000000,
        "iops_stddev" : 222.618681,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.600000,
      "sys_cpu" : 60.242857,
      "ctx" : 262841,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 41.038976,
        "50" : 57.387485,
        "100" : 1.520529,
        "250" : 0.049210,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1469749,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 22115,
      "write_ticks" : 0,
      "in_queue" : 22115,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87701: Tue Oct 25 23:41:28 2022
  read: IOPS=147k, BW=573MiB/s (600MB/s)(4009MiB/7001msec)
    slat (nsec): min=2174, max=178164, avg=2386.17, stdev=455.79
    clat (nsec): min=191, max=208732, avg=24370.69, stdev=8077.61
     lat (usec): min=11, max=217, avg=26.76, stdev= 8.17
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   16], 10.00th=[   17], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   29], 60.00th=[   29],
     | 70.00th=[   29], 80.00th=[   30], 90.00th=[   30], 95.00th=[   35],
     | 99.00th=[   53], 99.50th=[   55], 99.90th=[   84], 99.95th=[  100],
     | 99.99th=[  116]
   bw (  KiB/s): min=584704, max=588312, per=100.00%, avg=586556.71, stdev=891.19, samples=14
   iops        : min=146176, max=147078, avg=146639.00, stdev=222.62, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=41.04%, 50=57.39%, 100=1.52%
  lat (usec)   : 250=0.05%
  cpu          : usr=15.60%, sys=60.24%, ctx=262841, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1026222,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=573MiB/s (600MB/s), 573MiB/s-573MiB/s (600MB/s-600MB/s), io=4009MiB (4203MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1469749/0, merge=0/0, ticks=22115/0, in_queue=22115, util=99.09%

$ rm /tmp/fio-output_4K_4_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734102,
  "timestamp_ms" : 1666734102332,
  "time" : "Tue Oct 25 23:41:42 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4528324608,
        "io_kbytes" : 4422192,
        "bw_bytes" : 646811113,
        "bw" : 631651,
        "iops" : 157912.441080,
        "runtime" : 7001,
        "total_ios" : 1105545,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2275,
          "max" : 119033,
          "mean" : 2745.671780,
          "stddev" : 595.147245,
          "N" : 1105545
        },
        "clat_ns" : {
          "min" : 3747,
          "max" : 138570,
          "mean" : 22051.316446,
          "stddev" : 5281.925427,
          "N" : 1105548,
          "percentile" : {
            "1.000000" : 17792,
            "5.000000" : 19328,
            "10.000000" : 19328,
            "20.000000" : 19840,
            "30.000000" : 20864,
            "40.000000" : 20864,
            "50.000000" : 21120,
            "60.000000" : 21120,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 22400,
            "95.000000" : 33024,
            "99.000000" : 47360,
            "99.500000" : 50432,
            "99.900000" : 79360,
            "99.950000" : 93696,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 12774,
          "max" : 140935,
          "mean" : 24796.987255,
          "stddev" : 5227.303506,
          "N" : 1105548
        },
        "bw_min" : 630016,
        "bw_max" : 633688,
        "bw_agg" : 100.000000,
        "bw_mean" : 632033.142857,
        "bw_dev" : 1130.843651,
        "bw_samples" : 14,
        "iops_min" : 157504,
        "iops_max" : 158422,
        "iops_mean" : 158008.214286,
        "iops_stddev" : 282.708923,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.085714,
      "sys_cpu" : 57.100000,
      "ctx" : 284692,
      "majf" : 0,
      "minf" : 1,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 21.244273,
        "50" : 78.183339,
        "100" : 0.536839,
        "250" : 0.034915,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1564440,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 23344,
      "write_ticks" : 0,
      "in_queue" : 23344,
      "util" : 99.108381
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=87740: Tue Oct 25 23:41:42 2022
  read: IOPS=158k, BW=617MiB/s (647MB/s)(4319MiB/7001msec)
    slat (nsec): min=2275, max=119033, avg=2745.67, stdev=595.15
    clat (usec): min=3, max=138, avg=22.05, stdev= 5.28
     lat (usec): min=12, max=140, avg=24.80, stdev= 5.23
    clat percentiles (usec):
     |  1.00th=[   18],  5.00th=[   20], 10.00th=[   20], 20.00th=[   20],
     | 30.00th=[   21], 40.00th=[   21], 50.00th=[   22], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   23], 95.00th=[   34],
     | 99.00th=[   48], 99.50th=[   51], 99.90th=[   80], 99.95th=[   94],
     | 99.99th=[  106]
   bw (  KiB/s): min=630016, max=633688, per=100.00%, avg=632033.14, stdev=1130.84, samples=14
   iops        : min=157504, max=158422, avg=158008.21, stdev=282.71, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=21.24%, 50=78.18%, 100=0.54%
  lat (usec)   : 250=0.03%
  cpu          : usr=14.09%, sys=57.10%, ctx=284692, majf=0, minf=1
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1105545,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=617MiB/s (647MB/s), 617MiB/s-617MiB/s (647MB/s-647MB/s), io=4319MiB (4528MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1564440/0, merge=0/0, ticks=23344/0, in_queue=23344, util=99.11%

$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734117,
  "timestamp_ms" : 1666734117300,
  "time" : "Tue Oct 25 23:41:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 5844619264,
        "io_kbytes" : 5707636,
        "bw_bytes" : 834826348,
        "bw" : 815260,
        "iops" : 203814.597915,
        "runtime" : 7001,
        "total_ios" : 1426906,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 15208,
          "mean" : 166.282201,
          "stddev" : 53.441501,
          "N" : 1426906
        },
        "clat_ns" : {
          "min" : 10440,
          "max" : 145372,
          "mean" : 19068.696730,
          "stddev" : 5673.395713,
          "N" : 1426908,
          "percentile" : {
            "1.000000" : 15424,
            "5.000000" : 16320,
            "10.000000" : 17024,
            "20.000000" : 17536,
            "30.000000" : 17536,
            "40.000000" : 17536,
            "50.000000" : 17792,
            "60.000000" : 17792,
            "70.000000" : 18048,
            "80.000000" : 18048,
            "90.000000" : 19584,
            "95.000000" : 31104,
            "99.000000" : 43776,
            "99.500000" : 46336,
            "99.900000" : 83456,
            "99.950000" : 98816,
            "99.990000" : 123392
          }
        },
        "lat_ns" : {
          "min" : 10620,
          "max" : 145553,
          "mean" : 19234.978923,
          "stddev" : 5673.110075,
          "N" : 1426908
        },
        "bw_min" : 812432,
        "bw_max" : 821851,
        "bw_agg" : 100.000000,
        "bw_mean" : 815661.571429,
        "bw_dev" : 2674.021567,
        "bw_samples" : 14,
        "iops_min" : 203108,
        "iops_max" : 205462,
        "iops_mean" : 203915.428571,
        "iops_stddev" : 668.389306,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 60.214286,
      "sys_cpu" : 39.757143,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 30.973379,
        "4" : 69.014077,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 91.161156,
        "50" : 8.538124,
        "100" : 0.253556,
        "250" : 0.047305,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87788: Tue Oct 25 23:41:57 2022
  read: IOPS=204k, BW=796MiB/s (835MB/s)(5574MiB/7001msec)
    slat (nsec): min=110, max=15208, avg=166.28, stdev=53.44
    clat (usec): min=10, max=145, avg=19.07, stdev= 5.67
     lat (usec): min=10, max=145, avg=19.23, stdev= 5.67
    clat percentiles (usec):
     |  1.00th=[   16],  5.00th=[   17], 10.00th=[   18], 20.00th=[   18],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   18], 60.00th=[   18],
     | 70.00th=[   19], 80.00th=[   19], 90.00th=[   20], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   84], 99.95th=[   99],
     | 99.99th=[  124]
   bw (  KiB/s): min=812432, max=821851, per=100.00%, avg=815661.57, stdev=2674.02, samples=14
   iops        : min=203108, max=205462, avg=203915.43, stdev=668.39, samples=14
  lat (usec)   : 20=91.16%, 50=8.54%, 100=0.25%, 250=0.05%
  cpu          : usr=60.21%, sys=39.76%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=31.0%, 4=69.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1426906,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=796MiB/s (835MB/s), 796MiB/s-796MiB/s (835MB/s-835MB/s), io=5574MiB (5845MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733101,
  "timestamp_ms" : 1666733101115,
  "time" : "Tue Oct 25 23:25:01 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2411470848,
        "io_kbytes" : 2354952,
        "bw_bytes" : 344446628,
        "bw" : 336373,
        "iops" : 84093.415226,
        "runtime" : 7001,
        "total_ios" : 588738,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 25067,
          "mean" : 928.008393,
          "stddev" : 146.987262,
          "N" : 588738
        },
        "clat_ns" : {
          "min" : 701,
          "max" : 113202,
          "mean" : 10713.652358,
          "stddev" : 2206.439842,
          "N" : 588738,
          "percentile" : {
            "1.000000" : 9920,
            "5.000000" : 10048,
            "10.000000" : 10176,
            "20.000000" : 10176,
            "30.000000" : 10304,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10432,
            "70.000000" : 10432,
            "80.000000" : 10560,
            "90.000000" : 10688,
            "95.000000" : 11200,
            "99.000000" : 20096,
            "99.500000" : 27520,
            "99.900000" : 30592,
            "99.950000" : 43776,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 10590,
          "max" : 114104,
          "mean" : 11641.660751,
          "stddev" : 2210.286923,
          "N" : 588738
        },
        "bw_min" : 335992,
        "bw_max" : 336976,
        "bw_agg" : 100.000000,
        "bw_mean" : 336465.214286,
        "bw_dev" : 296.234361,
        "bw_samples" : 14,
        "iops_min" : 83998,
        "iops_max" : 84244,
        "iops_mean" : 84116.285714,
        "iops_stddev" : 74.034741,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 1.542282,
        "20" : 97.451498,
        "50" : 0.959000,
        "100" : 0.044842,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84683: Tue Oct 25 23:25:01 2022
  read: IOPS=84.1k, BW=328MiB/s (344MB/s)(2300MiB/7001msec)
    slat (nsec): min=711, max=25067, avg=928.01, stdev=146.99
    clat (nsec): min=701, max=113202, avg=10713.65, stdev=2206.44
     lat (usec): min=10, max=114, avg=11.64, stdev= 2.21
    clat percentiles (nsec):
     |  1.00th=[ 9920],  5.00th=[10048], 10.00th=[10176], 20.00th=[10176],
     | 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
     | 70.00th=[10432], 80.00th=[10560], 90.00th=[10688], 95.00th=[11200],
     | 99.00th=[20096], 99.50th=[27520], 99.90th=[30592], 99.95th=[43776],
     | 99.99th=[67072]
   bw (  KiB/s): min=335992, max=336976, per=100.00%, avg=336465.21, stdev=296.23, samples=14
   iops        : min=83998, max=84244, avg=84116.29, stdev=74.03, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=1.54%, 20=97.45%, 50=0.96%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=15, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=588738,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2300MiB (2411MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734132,
  "timestamp_ms" : 1666734132411,
  "time" : "Tue Oct 25 23:42:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6327066624,
        "io_kbytes" : 6178776,
        "bw_bytes" : 903737555,
        "bw" : 882556,
        "iops" : 220638.623054,
        "runtime" : 7001,
        "total_ios" : 1544691,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17432,
          "mean" : 189.121560,
          "stddev" : 121.269366,
          "N" : 1544691
        },
        "clat_ns" : {
          "min" : 9367,
          "max" : 202460,
          "mean" : 17679.470528,
          "stddev" : 5728.764872,
          "N" : 1544694,
          "percentile" : {
            "1.000000" : 11072,
            "5.000000" : 15296,
            "10.000000" : 15936,
            "20.000000" : 16064,
            "30.000000" : 16064,
            "40.000000" : 16064,
            "50.000000" : 16192,
            "60.000000" : 16192,
            "70.000000" : 16320,
            "80.000000" : 16512,
            "90.000000" : 20608,
            "95.000000" : 29312,
            "99.000000" : 39168,
            "99.500000" : 44288,
            "99.900000" : 81408,
            "99.950000" : 92672,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 9678,
          "max" : 202640,
          "mean" : 17868.592084,
          "stddev" : 5727.259130,
          "N" : 1544694
        },
        "bw_min" : 880480,
        "bw_max" : 884112,
        "bw_agg" : 100.000000,
        "bw_mean" : 882771.142857,
        "bw_dev" : 946.312914,
        "bw_samples" : 14,
        "iops_min" : 220120,
        "iops_max" : 221028,
        "iops_mean" : 220692.928571,
        "iops_stddev" : 236.341495,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 42.900000,
      "sys_cpu" : 57.071429,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.363309,
        "2" : 2.448580,
        "4" : 97.188111,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 89.582124,
        "50" : 10.162874,
        "100" : 0.218814,
        "250" : 0.032369,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87838: Tue Oct 25 23:42:12 2022
  read: IOPS=221k, BW=862MiB/s (904MB/s)(6034MiB/7001msec)
    slat (nsec): min=110, max=17432, avg=189.12, stdev=121.27
    clat (usec): min=9, max=202, avg=17.68, stdev= 5.73
     lat (usec): min=9, max=202, avg=17.87, stdev= 5.73
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   16], 10.00th=[   16], 20.00th=[   17],
     | 30.00th=[   17], 40.00th=[   17], 50.00th=[   17], 60.00th=[   17],
     | 70.00th=[   17], 80.00th=[   17], 90.00th=[   21], 95.00th=[   30],
     | 99.00th=[   40], 99.50th=[   45], 99.90th=[   82], 99.95th=[   93],
     | 99.99th=[  106]
   bw (  KiB/s): min=880480, max=884112, per=100.00%, avg=882771.14, stdev=946.31, samples=14
   iops        : min=220120, max=221028, avg=220692.93, stdev=236.34, samples=14
  lat (usec)   : 10=0.01%, 20=89.58%, 50=10.16%, 100=0.22%, 250=0.03%
  cpu          : usr=42.90%, sys=57.07%, ctx=11, majf=0, minf=0
  IO depths    : 1=0.4%, 2=2.4%, 4=97.2%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1544691,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=862MiB/s (904MB/s), 862MiB/s-862MiB/s (904MB/s-904MB/s), io=6034MiB (6327MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734147,
  "timestamp_ms" : 1666734147528,
  "time" : "Tue Oct 25 23:42:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6299238400,
        "io_kbytes" : 6151600,
        "bw_bytes" : 899762662,
        "bw" : 878674,
        "iops" : 219668.190259,
        "runtime" : 7001,
        "total_ios" : 1537897,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 120,
          "max" : 16481,
          "mean" : 214.052651,
          "stddev" : 212.685902,
          "N" : 1537897
        },
        "clat_ns" : {
          "min" : 9428,
          "max" : 196909,
          "mean" : 17698.782211,
          "stddev" : 5860.452451,
          "N" : 1537900,
          "percentile" : {
            "1.000000" : 11072,
            "5.000000" : 14144,
            "10.000000" : 15424,
            "20.000000" : 15936,
            "30.000000" : 16064,
            "40.000000" : 16064,
            "50.000000" : 16192,
            "60.000000" : 16192,
            "70.000000" : 16512,
            "80.000000" : 17280,
            "90.000000" : 21120,
            "95.000000" : 29568,
            "99.000000" : 41216,
            "99.500000" : 44288,
            "99.900000" : 81408,
            "99.950000" : 93696,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 9678,
          "max" : 197099,
          "mean" : 17912.834803,
          "stddev" : 5855.839323,
          "N" : 1537900
        },
        "bw_min" : 875880,
        "bw_max" : 881400,
        "bw_agg" : 100.000000,
        "bw_mean" : 878894.857143,
        "bw_dev" : 1748.287903,
        "bw_samples" : 14,
        "iops_min" : 218970,
        "iops_max" : 220350,
        "iops_mean" : 219723.714286,
        "iops_stddev" : 437.071976,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 43.414286,
      "sys_cpu" : 56.557143,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.328696,
        "2" : 5.216539,
        "4" : 94.454765,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 87.246285,
        "50" : 12.507144,
        "100" : 0.209442,
        "250" : 0.031602,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=87889: Tue Oct 25 23:42:27 2022
  read: IOPS=220k, BW=858MiB/s (900MB/s)(6007MiB/7001msec)
    slat (nsec): min=120, max=16481, avg=214.05, stdev=212.69
    clat (usec): min=9, max=196, avg=17.70, stdev= 5.86
     lat (usec): min=9, max=197, avg=17.91, stdev= 5.86
    clat percentiles (usec):
     |  1.00th=[   12],  5.00th=[   15], 10.00th=[   16], 20.00th=[   16],
     | 30.00th=[   17], 40.00th=[   17], 50.00th=[   17], 60.00th=[   17],
     | 70.00th=[   17], 80.00th=[   18], 90.00th=[   22], 95.00th=[   30],
     | 99.00th=[   42], 99.50th=[   45], 99.90th=[   82], 99.95th=[   94],
     | 99.99th=[  106]
   bw (  KiB/s): min=875880, max=881400, per=100.00%, avg=878894.86, stdev=1748.29, samples=14
   iops        : min=218970, max=220350, avg=219723.71, stdev=437.07, samples=14
  lat (usec)   : 10=0.01%, 20=87.25%, 50=12.51%, 100=0.21%, 250=0.03%
  cpu          : usr=43.41%, sys=56.56%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.3%, 2=5.2%, 4=94.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1537897,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=858MiB/s (900MB/s), 858MiB/s-858MiB/s (900MB/s-900MB/s), io=6007MiB (6299MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734162,
  "timestamp_ms" : 1666734162525,
  "time" : "Tue Oct 25 23:42:42 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6373732352,
        "io_kbytes" : 6224348,
        "bw_bytes" : 910403135,
        "bw" : 889065,
        "iops" : 222265.962005,
        "runtime" : 7001,
        "total_ios" : 1556084,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1462,
          "max" : 37179,
          "mean" : 1641.605073,
          "stddev" : 199.551002,
          "N" : 1556084
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 126758,
          "mean" : 16115.646010,
          "stddev" : 5418.188609,
          "N" : 1556087,
          "percentile" : {
            "1.000000" : 14144,
            "5.000000" : 14400,
            "10.000000" : 14528,
            "20.000000" : 14528,
            "30.000000" : 14656,
            "40.000000" : 14656,
            "50.000000" : 14656,
            "60.000000" : 14784,
            "70.000000" : 14784,
            "80.000000" : 14912,
            "90.000000" : 16512,
            "95.000000" : 27008,
            "99.000000" : 36096,
            "99.500000" : 42752,
            "99.900000" : 78336,
            "99.950000" : 88576,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 10790,
          "max" : 128381,
          "mean" : 17757.251002,
          "stddev" : 5418.749748,
          "N" : 1556087
        },
        "bw_min" : 888000,
        "bw_max" : 890464,
        "bw_agg" : 100.000000,
        "bw_mean" : 889269.357143,
        "bw_dev" : 942.858143,
        "bw_samples" : 14,
        "iops_min" : 222000,
        "iops_max" : 222616,
        "iops_mean" : 222317.285714,
        "iops_stddev" : 235.776435,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.328571,
      "sys_cpu" : 81.642857,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 91.798450,
        "50" : 8.025659,
        "100" : 0.146200,
        "250" : 0.019986,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87938: Tue Oct 25 23:42:42 2022
  read: IOPS=222k, BW=868MiB/s (910MB/s)(6078MiB/7001msec)
    slat (nsec): min=1462, max=37179, avg=1641.61, stdev=199.55
    clat (nsec): min=230, max=126758, avg=16115.65, stdev=5418.19
     lat (usec): min=10, max=128, avg=17.76, stdev= 5.42
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   15], 90.00th=[   17], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   43], 99.90th=[   79], 99.95th=[   89],
     | 99.99th=[  104]
   bw (  KiB/s): min=888000, max=890464, per=100.00%, avg=889269.36, stdev=942.86, samples=14
   iops        : min=222000, max=222616, avg=222317.29, stdev=235.78, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=91.80%, 50=8.03%, 100=0.15%
  lat (usec)   : 250=0.02%
  cpu          : usr=18.33%, sys=81.64%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1556084,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=868MiB/s (910MB/s), 868MiB/s-868MiB/s (910MB/s-910MB/s), io=6078MiB (6374MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734177,
  "timestamp_ms" : 1666734177491,
  "time" : "Tue Oct 25 23:42:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6295175168,
        "io_kbytes" : 6147632,
        "bw_bytes" : 899182283,
        "bw" : 878107,
        "iops" : 219526.496215,
        "runtime" : 7001,
        "total_ios" : 1536905,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1593,
          "max" : 27100,
          "mean" : 1862.124565,
          "stddev" : 155.390262,
          "N" : 1536905
        },
        "clat_ns" : {
          "min" : 250,
          "max" : 140985,
          "mean" : 16110.551588,
          "stddev" : 6695.143158,
          "N" : 1536908,
          "percentile" : {
            "1.000000" : 10432,
            "5.000000" : 10432,
            "10.000000" : 10560,
            "20.000000" : 12480,
            "30.000000" : 12736,
            "40.000000" : 13248,
            "50.000000" : 14400,
            "60.000000" : 14528,
            "70.000000" : 16768,
            "80.000000" : 20608,
            "90.000000" : 21376,
            "95.000000" : 29312,
            "99.000000" : 41216,
            "99.500000" : 44800,
            "99.900000" : 80384,
            "99.950000" : 96768,
            "99.990000" : 122368
          }
        },
        "lat_ns" : {
          "min" : 9629,
          "max" : 142838,
          "mean" : 17972.676267,
          "stddev" : 6706.536732,
          "N" : 1536908
        },
        "bw_min" : 876896,
        "bw_max" : 880945,
        "bw_agg" : 100.000000,
        "bw_mean" : 878672.428571,
        "bw_dev" : 1464.370472,
        "bw_samples" : 14,
        "iops_min" : 219224,
        "iops_max" : 220236,
        "iops_mean" : 219668.071429,
        "iops_stddev" : 366.064405,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.700000,
      "sys_cpu" : 82.271429,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.043334,
        "20" : 76.322609,
        "50" : 23.389604,
        "100" : 0.200533,
        "250" : 0.043919,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=87987: Tue Oct 25 23:42:57 2022
  read: IOPS=220k, BW=858MiB/s (899MB/s)(6004MiB/7001msec)
    slat (nsec): min=1593, max=27100, avg=1862.12, stdev=155.39
    clat (nsec): min=250, max=140985, avg=16110.55, stdev=6695.14
     lat (usec): min=9, max=142, avg=17.97, stdev= 6.71
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   11], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   14], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   17], 80.00th=[   21], 90.00th=[   22], 95.00th=[   30],
     | 99.00th=[   42], 99.50th=[   45], 99.90th=[   81], 99.95th=[   97],
     | 99.99th=[  123]
   bw (  KiB/s): min=876896, max=880945, per=100.00%, avg=878672.43, stdev=1464.37, samples=14
   iops        : min=219224, max=220236, avg=219668.07, stdev=366.06, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.04%, 20=76.32%, 50=23.39%, 100=0.20%, 250=0.04%
  cpu          : usr=17.70%, sys=82.27%, ctx=18, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1536905,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=858MiB/s (899MB/s), 858MiB/s-858MiB/s (899MB/s-899MB/s), io=6004MiB (6295MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ rm /tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734192,
  "timestamp_ms" : 1666734192584,
  "time" : "Tue Oct 25 23:43:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6370635776,
        "io_kbytes" : 6221324,
        "bw_bytes" : 909960830,
        "bw" : 888633,
        "iops" : 222157.834595,
        "runtime" : 7001,
        "total_ios" : 1555327,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1462,
          "max" : 27873,
          "mean" : 1640.823853,
          "stddev" : 204.088642,
          "N" : 1555328
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 140985,
          "mean" : 16120.417763,
          "stddev" : 5496.166937,
          "N" : 1555330,
          "percentile" : {
            "1.000000" : 14144,
            "5.000000" : 14400,
            "10.000000" : 14528,
            "20.000000" : 14528,
            "30.000000" : 14656,
            "40.000000" : 14656,
            "50.000000" : 14656,
            "60.000000" : 14784,
            "70.000000" : 14784,
            "80.000000" : 14912,
            "90.000000" : 16512,
            "95.000000" : 27008,
            "99.000000" : 36096,
            "99.500000" : 42752,
            "99.900000" : 79360,
            "99.950000" : 90624,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 10810,
          "max" : 142668,
          "mean" : 17761.241593,
          "stddev" : 5496.744254,
          "N" : 1555330
        },
        "bw_min" : 887288,
        "bw_max" : 890128,
        "bw_agg" : 100.000000,
        "bw_mean" : 888840.714286,
        "bw_dev" : 890.705375,
        "bw_samples" : 14,
        "iops_min" : 221822,
        "iops_max" : 222532,
        "iops_mean" : 222210.285714,
        "iops_stddev" : 222.724193,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.785714,
      "sys_cpu" : 81.185714,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.012152,
        "20" : 91.790730,
        "50" : 8.009377,
        "100" : 0.164274,
        "250" : 0.023082,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88037: Tue Oct 25 23:43:12 2022
  read: IOPS=222k, BW=868MiB/s (910MB/s)(6076MiB/7001msec)
    slat (nsec): min=1462, max=27873, avg=1640.82, stdev=204.09
    clat (nsec): min=240, max=140985, avg=16120.42, stdev=5496.17
     lat (usec): min=10, max=142, avg=17.76, stdev= 5.50
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   15], 90.00th=[   17], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   43], 99.90th=[   80], 99.95th=[   91],
     | 99.99th=[  104]
   bw (  KiB/s): min=887288, max=890128, per=100.00%, avg=888840.71, stdev=890.71, samples=14
   iops        : min=221822, max=222532, avg=222210.29, stdev=222.72, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=91.79%, 50=8.01%, 100=0.16%
  lat (usec)   : 250=0.02%
  cpu          : usr=18.79%, sys=81.19%, ctx=13, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1555327,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=868MiB/s (910MB/s), 868MiB/s-868MiB/s (910MB/s-910MB/s), io=6076MiB (6371MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734206,
  "timestamp_ms" : 1666734206301,
  "time" : "Tue Oct 25 23:43:26 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4069163008,
        "io_kbytes" : 3973792,
        "bw_bytes" : 581225968,
        "bw" : 567603,
        "iops" : 141900.442794,
        "runtime" : 7001,
        "total_ios" : 993445,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2274,
          "max" : 1343589,
          "mean" : 2510.766837,
          "stddev" : 1415.574549,
          "N" : 993445
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 1371542,
          "mean" : 25199.312395,
          "stddev" : 8781.217777,
          "N" : 993448,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 14272,
            "10.000000" : 14400,
            "20.000000" : 14656,
            "30.000000" : 20864,
            "40.000000" : 21376,
            "50.000000" : 29568,
            "60.000000" : 30080,
            "70.000000" : 30080,
            "80.000000" : 30336,
            "90.000000" : 30592,
            "95.000000" : 35072,
            "99.000000" : 52992,
            "99.500000" : 56576,
            "99.900000" : 82432,
            "99.950000" : 97792,
            "99.990000" : 114176
          }
        },
        "lat_ns" : {
          "min" : 14237,
          "max" : 1373906,
          "mean" : 27710.079092,
          "stddev" : 8975.141547,
          "N" : 993448
        },
        "bw_min" : 566520,
        "bw_max" : 570888,
        "bw_agg" : 100.000000,
        "bw_mean" : 567735.857143,
        "bw_dev" : 1371.074472,
        "bw_samples" : 14,
        "iops_min" : 141630,
        "iops_max" : 142722,
        "iops_mean" : 141933.928571,
        "iops_stddev" : 342.789118,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 20.185714,
      "sys_cpu" : 56.685714,
      "ctx" : 253738,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 24.128261,
        "50" : 74.194948,
        "100" : 1.629783,
        "250" : 0.044089,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88076: Tue Oct 25 23:43:26 2022
  read: IOPS=142k, BW=554MiB/s (581MB/s)(3881MiB/7001msec)
    slat (usec): min=2, max=1343, avg= 2.51, stdev= 1.42
    clat (nsec): min=240, max=1371.5k, avg=25199.31, stdev=8781.22
     lat (usec): min=14, max=1373, avg=27.71, stdev= 8.98
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   21], 40.00th=[   22], 50.00th=[   30], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   31], 90.00th=[   31], 95.00th=[   36],
     | 99.00th=[   53], 99.50th=[   57], 99.90th=[   83], 99.95th=[   98],
     | 99.99th=[  115]
   bw (  KiB/s): min=566520, max=570888, per=100.00%, avg=567735.86, stdev=1371.07, samples=14
   iops        : min=141630, max=142722, avg=141933.93, stdev=342.79, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=24.13%, 50=74.19%, 100=1.63%
  lat (usec)   : 250=0.04%
  lat (msec)   : 2=0.01%
  cpu          : usr=20.19%, sys=56.69%, ctx=253738, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=993445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=554MiB/s (581MB/s), 554MiB/s-554MiB/s (581MB/s-581MB/s), io=3881MiB (4069MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734220,
  "timestamp_ms" : 1666734220017,
  "time" : "Tue Oct 25 23:43:40 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4071936000,
        "io_kbytes" : 3976500,
        "bw_bytes" : 581622053,
        "bw" : 567990,
        "iops" : 141997.000429,
        "runtime" : 7001,
        "total_ios" : 994121,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2275,
          "max" : 47789,
          "mean" : 2511.122108,
          "stddev" : 454.419258,
          "N" : 994122
        },
        "clat_ns" : {
          "min" : 271,
          "max" : 165681,
          "mean" : 25181.048494,
          "stddev" : 8477.785485,
          "N" : 994124,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 14144,
            "10.000000" : 14272,
            "20.000000" : 14656,
            "30.000000" : 20864,
            "40.000000" : 21376,
            "50.000000" : 29568,
            "60.000000" : 30080,
            "70.000000" : 30080,
            "80.000000" : 30336,
            "90.000000" : 30592,
            "95.000000" : 34560,
            "99.000000" : 52992,
            "99.500000" : 56576,
            "99.900000" : 81408,
            "99.950000" : 96768,
            "99.990000" : 115200
          }
        },
        "lat_ns" : {
          "min" : 11572,
          "max" : 168056,
          "mean" : 27692.170811,
          "stddev" : 8578.284311,
          "N" : 994124
        },
        "bw_min" : 565416,
        "bw_max" : 571520,
        "bw_agg" : 100.000000,
        "bw_mean" : 568149.071429,
        "bw_dev" : 1825.775977,
        "bw_samples" : 14,
        "iops_min" : 141354,
        "iops_max" : 142880,
        "iops_mean" : 142037.214286,
        "iops_stddev" : 456.460661,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 18.328571,
      "sys_cpu" : 58.714286,
      "ctx" : 253915,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 24.695384,
        "50" : 73.607136,
        "100" : 1.651710,
        "250" : 0.043355,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88116: Tue Oct 25 23:43:40 2022
  read: IOPS=142k, BW=555MiB/s (582MB/s)(3883MiB/7001msec)
    slat (nsec): min=2275, max=47789, avg=2511.12, stdev=454.42
    clat (nsec): min=271, max=165681, avg=25181.05, stdev=8477.79
     lat (usec): min=11, max=168, avg=27.69, stdev= 8.58
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   21], 40.00th=[   22], 50.00th=[   30], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   31], 90.00th=[   31], 95.00th=[   35],
     | 99.00th=[   53], 99.50th=[   57], 99.90th=[   82], 99.95th=[   97],
     | 99.99th=[  116]
   bw (  KiB/s): min=565416, max=571520, per=100.00%, avg=568149.07, stdev=1825.78, samples=14
   iops        : min=141354, max=142880, avg=142037.21, stdev=456.46, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=24.70%, 50=73.61%, 100=1.65%
  lat (usec)   : 250=0.04%
  cpu          : usr=18.33%, sys=58.71%, ctx=253915, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=994121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=555MiB/s (582MB/s), 555MiB/s-555MiB/s (582MB/s-582MB/s), io=3883MiB (4072MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734233,
  "timestamp_ms" : 1666734233669,
  "time" : "Tue Oct 25 23:43:53 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 4061618176,
        "io_kbytes" : 3966424,
        "bw_bytes" : 580148289,
        "bw" : 566551,
        "iops" : 141637.194686,
        "runtime" : 7001,
        "total_ios" : 991602,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2294,
          "max" : 197260,
          "mean" : 2527.106294,
          "stddev" : 495.502687,
          "N" : 991603
        },
        "clat_ns" : {
          "min" : 230,
          "max" : 212398,
          "mean" : 25205.674751,
          "stddev" : 8552.840582,
          "N" : 991605,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 14144,
            "10.000000" : 14272,
            "20.000000" : 14528,
            "30.000000" : 20864,
            "40.000000" : 21120,
            "50.000000" : 29824,
            "60.000000" : 30080,
            "70.000000" : 30336,
            "80.000000" : 30336,
            "90.000000" : 30592,
            "95.000000" : 35072,
            "99.000000" : 52480,
            "99.500000" : 55552,
            "99.900000" : 84480,
            "99.950000" : 98816,
            "99.990000" : 116224
          }
        },
        "lat_ns" : {
          "min" : 11661,
          "max" : 215053,
          "mean" : 27732.780705,
          "stddev" : 8653.005043,
          "N" : 991605
        },
        "bw_min" : 565176,
        "bw_max" : 569450,
        "bw_agg" : 100.000000,
        "bw_mean" : 566683.571429,
        "bw_dev" : 1091.546871,
        "bw_samples" : 14,
        "iops_min" : 141294,
        "iops_max" : 142362,
        "iops_mean" : 141670.857143,
        "iops_stddev" : 272.789256,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.257143,
      "sys_cpu" : 55.942857,
      "ctx" : 253051,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 23.960924,
        "50" : 74.303501,
        "100" : 1.684345,
        "250" : 0.046793,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=88156: Tue Oct 25 23:43:53 2022
  read: IOPS=142k, BW=553MiB/s (580MB/s)(3873MiB/7001msec)
    slat (nsec): min=2294, max=197260, avg=2527.11, stdev=495.50
    clat (nsec): min=230, max=212398, avg=25205.67, stdev=8552.84
     lat (usec): min=11, max=215, avg=27.73, stdev= 8.65
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   21], 40.00th=[   22], 50.00th=[   30], 60.00th=[   31],
     | 70.00th=[   31], 80.00th=[   31], 90.00th=[   31], 95.00th=[   36],
     | 99.00th=[   53], 99.50th=[   56], 99.90th=[   85], 99.95th=[   99],
     | 99.99th=[  117]
   bw (  KiB/s): min=565176, max=569450, per=100.00%, avg=566683.57, stdev=1091.55, samples=14
   iops        : min=141294, max=142362, avg=141670.86, stdev=272.79, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=23.96%, 50=74.30%, 100=1.68%
  lat (usec)   : 250=0.05%
  cpu          : usr=21.26%, sys=55.94%, ctx=253051, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=991602,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=553MiB/s (580MB/s), 553MiB/s-553MiB/s (580MB/s-580MB/s), io=3873MiB (4062MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_4_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734247,
  "timestamp_ms" : 1666734247308,
  "time" : "Tue Oct 25 23:44:07 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5208461312,
        "io_kbytes" : 5086388,
        "bw_bytes" : 743959621,
        "bw" : 726523,
        "iops" : 181630.195686,
        "runtime" : 7001,
        "total_ios" : 1271593,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1994,
          "max" : 27552,
          "mean" : 2273.459399,
          "stddev" : 184.930489,
          "N" : 1271594
        },
        "clat_ns" : {
          "min" : 6773,
          "max" : 133701,
          "mean" : 19356.888477,
          "stddev" : 6100.168271,
          "N" : 1271596,
          "percentile" : {
            "1.000000" : 14400,
            "5.000000" : 14528,
            "10.000000" : 14656,
            "20.000000" : 14784,
            "30.000000" : 15168,
            "40.000000" : 15424,
            "50.000000" : 18560,
            "60.000000" : 21120,
            "70.000000" : 21120,
            "80.000000" : 21376,
            "90.000000" : 21888,
            "95.000000" : 30848,
            "99.000000" : 42240,
            "99.500000" : 47360,
            "99.900000" : 77312,
            "99.950000" : 91648,
            "99.990000" : 104960
          }
        },
        "lat_ns" : {
          "min" : 11512,
          "max" : 135855,
          "mean" : 21630.347846,
          "stddev" : 6127.066883,
          "N" : 1271596
        },
        "bw_min" : 724216,
        "bw_max" : 728976,
        "bw_agg" : 100.000000,
        "bw_mean" : 726728.285714,
        "bw_dev" : 1331.121877,
        "bw_samples" : 14,
        "iops_min" : 181054,
        "iops_max" : 182244,
        "iops_mean" : 181682.071429,
        "iops_stddev" : 332.780469,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.585714,
      "sys_cpu" : 48.000000,
      "ctx" : 330200,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 50.625082,
        "50" : 49.125703,
        "100" : 0.219253,
        "250" : 0.023750,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1806255,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 26402,
      "write_ticks" : 0,
      "in_queue" : 26402,
      "util" : 99.078934
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=88195: Tue Oct 25 23:44:07 2022
  read: IOPS=182k, BW=709MiB/s (744MB/s)(4967MiB/7001msec)
    slat (nsec): min=1994, max=27552, avg=2273.46, stdev=184.93
    clat (usec): min=6, max=133, avg=19.36, stdev= 6.10
     lat (usec): min=11, max=135, avg=21.63, stdev= 6.13
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   16], 40.00th=[   16], 50.00th=[   19], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   22], 95.00th=[   31],
     | 99.00th=[   43], 99.50th=[   48], 99.90th=[   78], 99.95th=[   92],
     | 99.99th=[  105]
   bw (  KiB/s): min=724216, max=728976, per=100.00%, avg=726728.29, stdev=1331.12, samples=14
   iops        : min=181054, max=182244, avg=181682.07, stdev=332.78, samples=14
  lat (usec)   : 10=0.01%, 20=50.63%, 50=49.13%, 100=0.22%, 250=0.02%
  cpu          : usr=21.59%, sys=48.00%, ctx=330200, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1271593,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=709MiB/s (744MB/s), 709MiB/s-709MiB/s (744MB/s-744MB/s), io=4967MiB (5208MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1806255/0, merge=0/0, ticks=26402/0, in_queue=26402, util=99.08%

$ rm /tmp/fio-output_4K_4_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_4_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734260,
  "timestamp_ms" : 1666734260967,
  "time" : "Tue Oct 25 23:44:20 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5129441280,
        "io_kbytes" : 5009220,
        "bw_bytes" : 732672658,
        "bw" : 715500,
        "iops" : 178874.589344,
        "runtime" : 7001,
        "total_ios" : 1252301,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1342,
          "max" : 38192,
          "mean" : 2214.863789,
          "stddev" : 1428.389632,
          "N" : 1252302
        },
        "clat_ns" : {
          "min" : 2564,
          "max" : 191189,
          "mean" : 19851.901731,
          "stddev" : 5831.909239,
          "N" : 1252304,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 15040,
            "10.000000" : 15552,
            "20.000000" : 15808,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 19072,
            "60.000000" : 19584,
            "70.000000" : 21376,
            "80.000000" : 21376,
            "90.000000" : 21888,
            "95.000000" : 31616,
            "99.000000" : 43264,
            "99.500000" : 46336,
            "99.900000" : 78336,
            "99.950000" : 92672,
            "99.990000" : 104960
          }
        },
        "lat_ns" : {
          "min" : 10289,
          "max" : 195036,
          "mean" : 22066.766055,
          "stddev" : 5810.190816,
          "N" : 1252304
        },
        "bw_min" : 705904,
        "bw_max" : 720104,
        "bw_agg" : 100.000000,
        "bw_mean" : 715783.785714,
        "bw_dev" : 3968.759430,
        "bw_samples" : 14,
        "iops_min" : 176476,
        "iops_max" : 180026,
        "iops_mean" : 178945.857143,
        "iops_stddev" : 992.094820,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.257143,
      "sys_cpu" : 45.128571,
      "ctx" : 319706,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.063084,
        "20" : 61.387717,
        "50" : 38.299898,
        "100" : 0.221991,
        "250" : 0.026990,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1786396,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 28771,
      "write_ticks" : 0,
      "in_queue" : 28771,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=88233: Tue Oct 25 23:44:20 2022
  read: IOPS=179k, BW=699MiB/s (733MB/s)(4892MiB/7001msec)
    slat (nsec): min=1342, max=38192, avg=2214.86, stdev=1428.39
    clat (usec): min=2, max=191, avg=19.85, stdev= 5.83
     lat (usec): min=10, max=195, avg=22.07, stdev= 5.81
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   16], 10.00th=[   16], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   22], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   79], 99.95th=[   93],
     | 99.99th=[  105]
   bw (  KiB/s): min=705904, max=720104, per=100.00%, avg=715783.79, stdev=3968.76, samples=14
   iops        : min=176476, max=180026, avg=178945.86, stdev=992.09, samples=14
  lat (usec)   : 4=0.01%, 10=0.06%, 20=61.39%, 50=38.30%, 100=0.22%
  lat (usec)   : 250=0.03%
  cpu          : usr=17.26%, sys=45.13%, ctx=319706, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1252301,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=699MiB/s (733MB/s), 699MiB/s-699MiB/s (733MB/s-733MB/s), io=4892MiB (5129MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1786396/0, merge=0/0, ticks=28771/0, in_queue=28771, util=99.09%

$ rm /tmp/fio-output_4K_4_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733116,
  "timestamp_ms" : 1666733116091,
  "time" : "Tue Oct 25 23:25:16 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2407424000,
        "io_kbytes" : 2351000,
        "bw_bytes" : 343868590,
        "bw" : 335809,
        "iops" : 83952.292530,
        "runtime" : 7001,
        "total_ios" : 587750,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 711,
          "max" : 11492,
          "mean" : 925.037659,
          "stddev" : 79.223115,
          "N" : 587750
        },
        "clat_ns" : {
          "min" : 3837,
          "max" : 176060,
          "mean" : 10698.287205,
          "stddev" : 2205.678474,
          "N" : 587750,
          "percentile" : {
            "1.000000" : 9920,
            "5.000000" : 10048,
            "10.000000" : 10048,
            "20.000000" : 10176,
            "30.000000" : 10304,
            "40.000000" : 10304,
            "50.000000" : 10432,
            "60.000000" : 10432,
            "70.000000" : 10560,
            "80.000000" : 10560,
            "90.000000" : 10688,
            "95.000000" : 11072,
            "99.000000" : 19840,
            "99.500000" : 27520,
            "99.900000" : 28032,
            "99.950000" : 43776,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 10570,
          "max" : 179687,
          "mean" : 11623.324864,
          "stddev" : 2208.087793,
          "N" : 587750
        },
        "bw_min" : 335496,
        "bw_max" : 336312,
        "bw_agg" : 100.000000,
        "bw_mean" : 335897.642857,
        "bw_dev" : 239.573213,
        "bw_samples" : 14,
        "iops_min" : 83874,
        "iops_max" : 84078,
        "iops_mean" : 83974.357143,
        "iops_stddev" : 59.925091,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 20,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 3.303956,
        "20" : 95.712123,
        "50" : 0.938494,
        "100" : 0.044747,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=84733: Tue Oct 25 23:25:16 2022
  read: IOPS=84.0k, BW=328MiB/s (344MB/s)(2296MiB/7001msec)
    slat (nsec): min=711, max=11492, avg=925.04, stdev=79.22
    clat (usec): min=3, max=176, avg=10.70, stdev= 2.21
     lat (usec): min=10, max=179, avg=11.62, stdev= 2.21
    clat percentiles (nsec):
     |  1.00th=[ 9920],  5.00th=[10048], 10.00th=[10048], 20.00th=[10176],
     | 30.00th=[10304], 40.00th=[10304], 50.00th=[10432], 60.00th=[10432],
     | 70.00th=[10560], 80.00th=[10560], 90.00th=[10688], 95.00th=[11072],
     | 99.00th=[19840], 99.50th=[27520], 99.90th=[28032], 99.95th=[43776],
     | 99.99th=[67072]
   bw (  KiB/s): min=335496, max=336312, per=100.00%, avg=335897.64, stdev=239.57, samples=14
   iops        : min=83874, max=84078, avg=83974.36, stdev=59.93, samples=14
  lat (usec)   : 4=0.01%, 10=3.30%, 20=95.71%, 50=0.94%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=99.99%, sys=0.00%, ctx=20, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=587750,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2296MiB (2407MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_4_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734274,
  "timestamp_ms" : 1666734274606,
  "time" : "Tue Oct 25 23:44:34 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5094621184,
        "io_kbytes" : 4975216,
        "bw_bytes" : 727699069,
        "bw" : 710643,
        "iops" : 177660.477075,
        "runtime" : 7001,
        "total_ios" : 1243801,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1322,
          "max" : 179357,
          "mean" : 2270.165353,
          "stddev" : 1482.663537,
          "N" : 1243801
        },
        "clat_ns" : {
          "min" : 2415,
          "max" : 1314775,
          "mean" : 19921.951998,
          "stddev" : 6421.548737,
          "N" : 1243804,
          "percentile" : {
            "1.000000" : 14016,
            "5.000000" : 15040,
            "10.000000" : 15552,
            "20.000000" : 15808,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 19072,
            "60.000000" : 19328,
            "70.000000" : 21120,
            "80.000000" : 21376,
            "90.000000" : 23680,
            "95.000000" : 31872,
            "99.000000" : 43264,
            "99.500000" : 47360,
            "99.900000" : 79360,
            "99.950000" : 93696,
            "99.990000" : 104960
          }
        },
        "lat_ns" : {
          "min" : 9989,
          "max" : 1316138,
          "mean" : 22192.118014,
          "stddev" : 6367.348917,
          "N" : 1243804
        },
        "bw_min" : 653496,
        "bw_max" : 722680,
        "bw_agg" : 100.000000,
        "bw_mean" : 710909.000000,
        "bw_dev" : 18576.458195,
        "bw_samples" : 14,
        "iops_min" : 163374,
        "iops_max" : 180670,
        "iops_mean" : 177727.285714,
        "iops_stddev" : 4644.186298,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.771429,
      "sys_cpu" : 45.885714,
      "ctx" : 313922,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.049365,
        "20" : 62.858367,
        "50" : 36.778311,
        "100" : 0.285978,
        "250" : 0.027336,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 1774792,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 28485,
      "write_ticks" : 0,
      "in_queue" : 28485,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=88271: Tue Oct 25 23:44:34 2022
  read: IOPS=178k, BW=694MiB/s (728MB/s)(4859MiB/7001msec)
    slat (nsec): min=1322, max=179357, avg=2270.17, stdev=1482.66
    clat (usec): min=2, max=1314, avg=19.92, stdev= 6.42
     lat (usec): min=9, max=1316, avg=22.19, stdev= 6.37
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   16], 10.00th=[   16], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   24], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   48], 99.90th=[   80], 99.95th=[   94],
     | 99.99th=[  105]
   bw (  KiB/s): min=653496, max=722680, per=100.00%, avg=710909.00, stdev=18576.46, samples=14
   iops        : min=163374, max=180670, avg=177727.29, stdev=4644.19, samples=14
  lat (usec)   : 4=0.01%, 10=0.05%, 20=62.86%, 50=36.78%, 100=0.29%
  lat (usec)   : 250=0.03%
  lat (msec)   : 2=0.01%
  cpu          : usr=17.77%, sys=45.89%, ctx=313922, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1243801,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=694MiB/s (728MB/s), 694MiB/s-694MiB/s (728MB/s-728MB/s), io=4859MiB (5095MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=1774792/0, merge=0/0, ticks=28485/0, in_queue=28485, util=99.09%

$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734289,
  "timestamp_ms" : 1666734289535,
  "time" : "Tue Oct 25 23:44:49 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6368309248,
        "io_kbytes" : 6219052,
        "bw_bytes" : 909628517,
        "bw" : 888309,
        "iops" : 222076.846165,
        "runtime" : 7001,
        "total_ios" : 1554760,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1523,
          "max" : 20198,
          "mean" : 1636.536629,
          "stddev" : 223.818184,
          "N" : 1554760
        },
        "clat_ns" : {
          "min" : 251,
          "max" : 148058,
          "mean" : 16084.288581,
          "stddev" : 5476.140778,
          "N" : 1554763,
          "percentile" : {
            "1.000000" : 13504,
            "5.000000" : 14400,
            "10.000000" : 14528,
            "20.000000" : 14528,
            "30.000000" : 14656,
            "40.000000" : 14656,
            "50.000000" : 14656,
            "60.000000" : 14784,
            "70.000000" : 14784,
            "80.000000" : 14912,
            "90.000000" : 16512,
            "95.000000" : 27008,
            "99.000000" : 36096,
            "99.500000" : 42752,
            "99.900000" : 79360,
            "99.950000" : 90624,
            "99.990000" : 104960
          }
        },
        "lat_ns" : {
          "min" : 9618,
          "max" : 149671,
          "mean" : 17720.825126,
          "stddev" : 5474.918573,
          "N" : 1554763
        },
        "bw_min" : 886712,
        "bw_max" : 890216,
        "bw_agg" : 100.000000,
        "bw_mean" : 888525.785714,
        "bw_dev" : 1043.348321,
        "bw_samples" : 14,
        "iops_min" : 221678,
        "iops_max" : 222554,
        "iops_mean" : 222131.428571,
        "iops_stddev" : 260.832311,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 73.700000,
      "sys_cpu" : 26.285714,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.372984,
        "2" : 0.960470,
        "4" : 98.666547,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.063933,
        "20" : 92.064048,
        "50" : 7.686588,
        "100" : 0.157516,
        "250" : 0.025663,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88319: Tue Oct 25 23:44:49 2022
  read: IOPS=222k, BW=867MiB/s (910MB/s)(6073MiB/7001msec)
    slat (nsec): min=1523, max=20198, avg=1636.54, stdev=223.82
    clat (nsec): min=251, max=148058, avg=16084.29, stdev=5476.14
     lat (usec): min=9, max=149, avg=17.72, stdev= 5.47
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   15], 90.00th=[   17], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   43], 99.90th=[   80], 99.95th=[   91],
     | 99.99th=[  105]
   bw (  KiB/s): min=886712, max=890216, per=100.00%, avg=888525.79, stdev=1043.35, samples=14
   iops        : min=221678, max=222554, avg=222131.43, stdev=260.83, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.06%, 20=92.06%, 50=7.69%, 100=0.16%
  lat (usec)   : 250=0.03%
  cpu          : usr=73.70%, sys=26.29%, ctx=4, majf=0, minf=0
  IO depths    : 1=0.4%, 2=1.0%, 4=98.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1554760,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=867MiB/s (910MB/s), 867MiB/s-867MiB/s (910MB/s-910MB/s), io=6073MiB (6368MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734304,
  "timestamp_ms" : 1666734304702,
  "time" : "Tue Oct 25 23:45:04 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6271488000,
        "io_kbytes" : 6124500,
        "bw_bytes" : 895798885,
        "bw" : 874803,
        "iops" : 218700.328524,
        "runtime" : 7001,
        "total_ios" : 1531121,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 62267,
          "mean" : 1883.836522,
          "stddev" : 167.538851,
          "N" : 1531122
        },
        "clat_ns" : {
          "min" : 290,
          "max" : 141535,
          "mean" : 16162.363481,
          "stddev" : 6497.223878,
          "N" : 1531121,
          "percentile" : {
            "1.000000" : 10432,
            "5.000000" : 10432,
            "10.000000" : 10560,
            "20.000000" : 12480,
            "30.000000" : 12864,
            "40.000000" : 14272,
            "50.000000" : 14528,
            "60.000000" : 14784,
            "70.000000" : 17024,
            "80.000000" : 19328,
            "90.000000" : 21376,
            "95.000000" : 28800,
            "99.000000" : 41216,
            "99.500000" : 44800,
            "99.900000" : 77312,
            "99.950000" : 94720,
            "99.990000" : 121344
          }
        },
        "lat_ns" : {
          "min" : 9808,
          "max" : 143308,
          "mean" : 18046.199945,
          "stddev" : 6499.210762,
          "N" : 1531121
        },
        "bw_min" : 873272,
        "bw_max" : 877680,
        "bw_agg" : 100.000000,
        "bw_mean" : 875229.571429,
        "bw_dev" : 1466.632654,
        "bw_samples" : 14,
        "iops_min" : 218318,
        "iops_max" : 219420,
        "iops_mean" : 218807.285714,
        "iops_stddev" : 366.406758,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 77.814286,
      "sys_cpu" : 22.157143,
      "ctx" : 22,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.441180,
        "4" : 99.548239,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.063418,
        "20" : 80.184910,
        "50" : 19.526347,
        "100" : 0.182611,
        "250" : 0.041016,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88368: Tue Oct 25 23:45:04 2022
  read: IOPS=219k, BW=854MiB/s (896MB/s)(5981MiB/7001msec)
    slat (nsec): min=1643, max=62267, avg=1883.84, stdev=167.54
    clat (nsec): min=290, max=141535, avg=16162.36, stdev=6497.22
     lat (usec): min=9, max=143, avg=18.05, stdev= 6.50
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   11], 20.00th=[   13],
     | 30.00th=[   13], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   18], 80.00th=[   20], 90.00th=[   22], 95.00th=[   29],
     | 99.00th=[   42], 99.50th=[   45], 99.90th=[   78], 99.95th=[   95],
     | 99.99th=[  122]
   bw (  KiB/s): min=873272, max=877680, per=100.00%, avg=875229.57, stdev=1466.63, samples=14
   iops        : min=218318, max=219420, avg=218807.29, stdev=366.41, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.06%, 20=80.18%, 50=19.53%, 100=0.18%, 250=0.04%
  cpu          : usr=77.81%, sys=22.16%, ctx=22, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.4%, 4=99.5%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1531121,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=854MiB/s (896MB/s), 854MiB/s-854MiB/s (896MB/s-896MB/s), io=5981MiB (6271MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734319,
  "timestamp_ms" : 1666734319976,
  "time" : "Tue Oct 25 23:45:19 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6364598272,
        "io_kbytes" : 6215428,
        "bw_bytes" : 909098453,
        "bw" : 887791,
        "iops" : 221947.293244,
        "runtime" : 7001,
        "total_ios" : 1553853,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1513,
          "max" : 25969,
          "mean" : 1631.607934,
          "stddev" : 243.107289,
          "N" : 1553854
        },
        "clat_ns" : {
          "min" : 261,
          "max" : 131006,
          "mean" : 16099.061237,
          "stddev" : 5450.853875,
          "N" : 1553853,
          "percentile" : {
            "1.000000" : 13504,
            "5.000000" : 14400,
            "10.000000" : 14528,
            "20.000000" : 14528,
            "30.000000" : 14656,
            "40.000000" : 14656,
            "50.000000" : 14656,
            "60.000000" : 14784,
            "70.000000" : 14784,
            "80.000000" : 14912,
            "90.000000" : 16512,
            "95.000000" : 27008,
            "99.000000" : 36096,
            "99.500000" : 42752,
            "99.900000" : 79360,
            "99.950000" : 88576,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 9708,
          "max" : 132579,
          "mean" : 17730.669197,
          "stddev" : 5450.382413,
          "N" : 1553853
        },
        "bw_min" : 886560,
        "bw_max" : 889792,
        "bw_agg" : 100.000000,
        "bw_mean" : 888029.857143,
        "bw_dev" : 1001.853195,
        "bw_samples" : 14,
        "iops_min" : 221640,
        "iops_max" : 222448,
        "iops_mean" : 222007.428571,
        "iops_stddev" : 250.424168,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 69.928571,
      "sys_cpu" : 30.042857,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.372300,
        "2" : 0.969268,
        "4" : 98.658432,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.058757,
        "20" : 91.962818,
        "50" : 7.797906,
        "100" : 0.153618,
        "250" : 0.024327,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=88418: Tue Oct 25 23:45:19 2022
  read: IOPS=222k, BW=867MiB/s (909MB/s)(6070MiB/7001msec)
    slat (nsec): min=1513, max=25969, avg=1631.61, stdev=243.11
    clat (nsec): min=261, max=131006, avg=16099.06, stdev=5450.85
     lat (usec): min=9, max=132, avg=17.73, stdev= 5.45
    clat percentiles (usec):
     |  1.00th=[   14],  5.00th=[   15], 10.00th=[   15], 20.00th=[   15],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   15], 90.00th=[   17], 95.00th=[   28],
     | 99.00th=[   37], 99.50th=[   43], 99.90th=[   80], 99.95th=[   89],
     | 99.99th=[  104]
   bw (  KiB/s): min=886560, max=889792, per=100.00%, avg=888029.86, stdev=1001.85, samples=14
   iops        : min=221640, max=222448, avg=222007.43, stdev=250.42, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 10=0.06%, 20=91.96%, 50=7.80%, 100=0.15%, 250=0.02%
  cpu          : usr=69.93%, sys=30.04%, ctx=4, majf=0, minf=0
  IO depths    : 1=0.4%, 2=1.0%, 4=98.7%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1553853,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=867MiB/s (909MB/s), 867MiB/s-867MiB/s (909MB/s-909MB/s), io=6070MiB (6365MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_0.txt': No such file or directory

$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734335,
  "timestamp_ms" : 1666734335155,
  "time" : "Tue Oct 25 23:45:35 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6271553536,
        "io_kbytes" : 6124564,
        "bw_bytes" : 895808246,
        "bw" : 874812,
        "iops" : 218702.756749,
        "runtime" : 7001,
        "total_ios" : 1531138,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1552,
          "max" : 172423,
          "mean" : 2025.800648,
          "stddev" : 791.208125,
          "N" : 1531138
        },
        "clat_ns" : {
          "min" : 651,
          "max" : 183304,
          "mean" : 15858.742410,
          "stddev" : 5803.263051,
          "N" : 1531141,
          "percentile" : {
            "1.000000" : 10176,
            "5.000000" : 10560,
            "10.000000" : 12992,
            "20.000000" : 13760,
            "30.000000" : 14016,
            "40.000000" : 14272,
            "50.000000" : 14528,
            "60.000000" : 14784,
            "70.000000" : 15168,
            "80.000000" : 16192,
            "90.000000" : 17792,
            "95.000000" : 27520,
            "99.000000" : 38656,
            "99.500000" : 42752,
            "99.900000" : 79360,
            "99.950000" : 90624,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 9658,
          "max" : 186690,
          "mean" : 17884.542243,
          "stddev" : 5775.342587,
          "N" : 1531141
        },
        "bw_min" : 872848,
        "bw_max" : 876888,
        "bw_agg" : 100.000000,
        "bw_mean" : 875033.785714,
        "bw_dev" : 1002.540326,
        "bw_samples" : 14,
        "iops_min" : 218212,
        "iops_max" : 219222,
        "iops_mean" : 218758.428571,
        "iops_stddev" : 250.748024,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 42.671429,
      "sys_cpu" : 57.300000,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.273522,
        "2" : 6.079334,
        "4" : 93.647143,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.020834,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.145970,
        "20" : 92.134086,
        "50" : 7.513366,
        "100" : 0.155897,
        "250" : 0.024753,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88468: Tue Oct 25 23:45:35 2022
  read: IOPS=219k, BW=854MiB/s (896MB/s)(5981MiB/7001msec)
    slat (nsec): min=1552, max=172423, avg=2025.80, stdev=791.21
    clat (nsec): min=651, max=183304, avg=15858.74, stdev=5803.26
     lat (usec): min=9, max=186, avg=17.88, stdev= 5.78
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   13], 20.00th=[   14],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   16], 80.00th=[   17], 90.00th=[   18], 95.00th=[   28],
     | 99.00th=[   39], 99.50th=[   43], 99.90th=[   80], 99.95th=[   91],
     | 99.99th=[  104]
   bw (  KiB/s): min=872848, max=876888, per=100.00%, avg=875033.79, stdev=1002.54, samples=14
   iops        : min=218212, max=219222, avg=218758.43, stdev=250.75, samples=14
  lat (nsec)   : 750=0.02%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.15%, 20=92.13%, 50=7.51%, 100=0.16%
  lat (usec)   : 250=0.02%
  cpu          : usr=42.67%, sys=57.30%, ctx=12, majf=0, minf=0
  IO depths    : 1=0.3%, 2=6.1%, 4=93.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1531138,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=854MiB/s (896MB/s), 854MiB/s-854MiB/s (896MB/s-896MB/s), io=5981MiB (6272MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734350,
  "timestamp_ms" : 1666734350304,
  "time" : "Tue Oct 25 23:45:50 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6267457536,
        "io_kbytes" : 6120564,
        "bw_bytes" : 895223187,
        "bw" : 874241,
        "iops" : 218559.777175,
        "runtime" : 7001,
        "total_ios" : 1530137,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1532,
          "max" : 20007,
          "mean" : 2014.375794,
          "stddev" : 757.866366,
          "N" : 1530138
        },
        "clat_ns" : {
          "min" : 641,
          "max" : 144321,
          "mean" : 15884.766559,
          "stddev" : 5830.698008,
          "N" : 1530137,
          "percentile" : {
            "1.000000" : 10176,
            "5.000000" : 10560,
            "10.000000" : 12992,
            "20.000000" : 13760,
            "30.000000" : 14016,
            "40.000000" : 14272,
            "50.000000" : 14528,
            "60.000000" : 14784,
            "70.000000" : 15168,
            "80.000000" : 16192,
            "90.000000" : 18048,
            "95.000000" : 27520,
            "99.000000" : 38656,
            "99.500000" : 42752,
            "99.900000" : 80384,
            "99.950000" : 90624,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 9969,
          "max" : 145914,
          "mean" : 17899.142622,
          "stddev" : 5795.903965,
          "N" : 1530137
        },
        "bw_min" : 871464,
        "bw_max" : 876000,
        "bw_agg" : 100.000000,
        "bw_mean" : 874451.071429,
        "bw_dev" : 1455.933772,
        "bw_samples" : 14,
        "iops_min" : 217866,
        "iops_max" : 219000,
        "iops_mean" : 218612.857143,
        "iops_stddev" : 364.172161,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.014286,
      "sys_cpu" : 58.957143,
      "ctx" : 6,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.280106,
        "2" : 6.138274,
        "4" : 93.581620,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.018953,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.173122,
        "20" : 92.036465,
        "50" : 7.573636,
        "100" : 0.166521,
        "250" : 0.026403,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88518: Tue Oct 25 23:45:50 2022
  read: IOPS=219k, BW=854MiB/s (895MB/s)(5977MiB/7001msec)
    slat (nsec): min=1532, max=20007, avg=2014.38, stdev=757.87
    clat (nsec): min=641, max=144321, avg=15884.77, stdev=5830.70
     lat (usec): min=9, max=145, avg=17.90, stdev= 5.80
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   13], 20.00th=[   14],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   16], 80.00th=[   17], 90.00th=[   19], 95.00th=[   28],
     | 99.00th=[   39], 99.50th=[   43], 99.90th=[   81], 99.95th=[   91],
     | 99.99th=[  104]
   bw (  KiB/s): min=871464, max=876000, per=100.00%, avg=874451.07, stdev=1455.93, samples=14
   iops        : min=217866, max=219000, avg=218612.86, stdev=364.17, samples=14
  lat (nsec)   : 750=0.02%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.17%, 20=92.04%, 50=7.57%, 100=0.17%
  lat (usec)   : 250=0.03%
  cpu          : usr=41.01%, sys=58.96%, ctx=6, majf=0, minf=0
  IO depths    : 1=0.3%, 2=6.1%, 4=93.6%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1530137,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=854MiB/s (895MB/s), 854MiB/s-854MiB/s (895MB/s-895MB/s), io=5977MiB (6267MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_4_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734365,
  "timestamp_ms" : 1666734365443,
  "time" : "Tue Oct 25 23:46:05 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6263152640,
        "io_kbytes" : 6116360,
        "bw_bytes" : 894608290,
        "bw" : 873640,
        "iops" : 218409.798600,
        "runtime" : 7001,
        "total_ios" : 1529087,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1552,
          "max" : 1296060,
          "mean" : 2038.297024,
          "stddev" : 1293.969014,
          "N" : 1529087
        },
        "clat_ns" : {
          "min" : 652,
          "max" : 1309395,
          "mean" : 15867.342854,
          "stddev" : 6045.816309,
          "N" : 1529090,
          "percentile" : {
            "1.000000" : 10304,
            "5.000000" : 10560,
            "10.000000" : 12992,
            "20.000000" : 13760,
            "30.000000" : 14016,
            "40.000000" : 14400,
            "50.000000" : 14656,
            "60.000000" : 14784,
            "70.000000" : 15168,
            "80.000000" : 16192,
            "90.000000" : 18048,
            "95.000000" : 27520,
            "99.000000" : 38656,
            "99.500000" : 42752,
            "99.900000" : 78336,
            "99.950000" : 90624,
            "99.990000" : 103936
          }
        },
        "lat_ns" : {
          "min" : 9738,
          "max" : 1311068,
          "mean" : 17905.639056,
          "stddev" : 6101.138943,
          "N" : 1529090
        },
        "bw_min" : 871208,
        "bw_max" : 876849,
        "bw_agg" : 100.000000,
        "bw_mean" : 873847.500000,
        "bw_dev" : 1583.292322,
        "bw_samples" : 14,
        "iops_min" : 217802,
        "iops_max" : 219212,
        "iops_mean" : 218461.857143,
        "iops_stddev" : 395.786628,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.414286,
      "sys_cpu" : 58.557143,
      "ctx" : 45,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.279579,
        "2" : 6.606622,
        "4" : 93.113799,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.020470,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.132170,
        "20" : 92.125039,
        "50" : 7.538289,
        "100" : 0.154994,
        "250" : 0.023413,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88567: Tue Oct 25 23:46:05 2022
  read: IOPS=218k, BW=853MiB/s (895MB/s)(5973MiB/7001msec)
    slat (nsec): min=1552, max=1296.1k, avg=2038.30, stdev=1293.97
    clat (nsec): min=652, max=1309.4k, avg=15867.34, stdev=6045.82
     lat (usec): min=9, max=1311, avg=17.91, stdev= 6.10
    clat percentiles (usec):
     |  1.00th=[   11],  5.00th=[   11], 10.00th=[   13], 20.00th=[   14],
     | 30.00th=[   15], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   16], 80.00th=[   17], 90.00th=[   19], 95.00th=[   28],
     | 99.00th=[   39], 99.50th=[   43], 99.90th=[   79], 99.95th=[   91],
     | 99.99th=[  104]
   bw (  KiB/s): min=871208, max=876849, per=100.00%, avg=873847.50, stdev=1583.29, samples=14
   iops        : min=217802, max=219212, avg=218461.86, stdev=395.79, samples=14
  lat (nsec)   : 750=0.02%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=0.13%, 20=92.13%, 50=7.54%, 100=0.15%
  lat (usec)   : 250=0.02%
  lat (msec)   : 2=0.01%
  cpu          : usr=41.41%, sys=58.56%, ctx=45, majf=0, minf=0
  IO depths    : 1=0.3%, 2=6.6%, 4=93.1%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1529087,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=853MiB/s (895MB/s), 853MiB/s-853MiB/s (895MB/s-895MB/s), io=5973MiB (6263MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_0.txt"
$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734379,
  "timestamp_ms" : 1666734379201,
  "time" : "Tue Oct 25 23:46:19 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5072072704,
        "io_kbytes" : 4953196,
        "bw_bytes" : 724478317,
        "bw" : 707498,
        "iops" : 176874.160834,
        "runtime" : 7001,
        "total_ios" : 1238296,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1392,
          "max" : 24916,
          "mean" : 2289.736705,
          "stddev" : 1489.650647,
          "N" : 1238296
        },
        "clat_ns" : {
          "min" : 2695,
          "max" : 133280,
          "mean" : 20010.147292,
          "stddev" : 5825.442065,
          "N" : 1238299,
          "percentile" : {
            "1.000000" : 14912,
            "5.000000" : 15680,
            "10.000000" : 15808,
            "20.000000" : 16064,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 19328,
            "60.000000" : 19584,
            "70.000000" : 21376,
            "80.000000" : 21888,
            "90.000000" : 22144,
            "95.000000" : 31360,
            "99.000000" : 43776,
            "99.500000" : 46848,
            "99.900000" : 81408,
            "99.950000" : 93696,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 10369,
          "max" : 134742,
          "mean" : 22299.882204,
          "stddev" : 5670.530977,
          "N" : 1238299
        },
        "bw_min" : 703320,
        "bw_max" : 710557,
        "bw_agg" : 100.000000,
        "bw_mean" : 707687.785714,
        "bw_dev" : 1699.738812,
        "bw_samples" : 14,
        "iops_min" : 175830,
        "iops_max" : 177639,
        "iops_mean" : 176921.928571,
        "iops_stddev" : 424.902245,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.414286,
      "sys_cpu" : 50.042857,
      "ctx" : 319314,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.027215,
        "20" : 65.634549,
        "50" : 34.085227,
        "100" : 0.224744,
        "250" : 0.028345,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88606: Tue Oct 25 23:46:19 2022
  read: IOPS=177k, BW=691MiB/s (724MB/s)(4837MiB/7001msec)
    slat (nsec): min=1392, max=24916, avg=2289.74, stdev=1489.65
    clat (usec): min=2, max=133, avg=20.01, stdev= 5.83
     lat (usec): min=10, max=134, avg=22.30, stdev= 5.67
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   16], 10.00th=[   16], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   23], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   82], 99.95th=[   94],
     | 99.99th=[  106]
   bw (  KiB/s): min=703320, max=710557, per=100.00%, avg=707687.79, stdev=1699.74, samples=14
   iops        : min=175830, max=177639, avg=176921.93, stdev=424.90, samples=14
  lat (usec)   : 4=0.01%, 10=0.03%, 20=65.63%, 50=34.09%, 100=0.22%
  lat (usec)   : 250=0.03%
  cpu          : usr=15.41%, sys=50.04%, ctx=319314, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1238296,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=691MiB/s (724MB/s), 691MiB/s-691MiB/s (724MB/s-724MB/s), io=4837MiB (5072MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734392,
  "timestamp_ms" : 1666734392857,
  "time" : "Tue Oct 25 23:46:32 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5079027712,
        "io_kbytes" : 4959988,
        "bw_bytes" : 725471748,
        "bw" : 708468,
        "iops" : 177116.554778,
        "runtime" : 7001,
        "total_ios" : 1239993,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1432,
          "max" : 25278,
          "mean" : 2296.353298,
          "stddev" : 1460.013791,
          "N" : 1239994
        },
        "clat_ns" : {
          "min" : 5000,
          "max" : 315411,
          "mean" : 19955.888521,
          "stddev" : 5758.486187,
          "N" : 1239996,
          "percentile" : {
            "1.000000" : 14656,
            "5.000000" : 15808,
            "10.000000" : 15808,
            "20.000000" : 16064,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 19328,
            "60.000000" : 19584,
            "70.000000" : 21376,
            "80.000000" : 21632,
            "90.000000" : 21888,
            "95.000000" : 31360,
            "99.000000" : 43264,
            "99.500000" : 46336,
            "99.900000" : 76288,
            "99.950000" : 92672,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 10500,
          "max" : 316904,
          "mean" : 22252.243399,
          "stddev" : 5586.706939,
          "N" : 1239996
        },
        "bw_min" : 704384,
        "bw_max" : 710608,
        "bw_agg" : 100.000000,
        "bw_mean" : 708637.857143,
        "bw_dev" : 1613.810486,
        "bw_samples" : 14,
        "iops_min" : 176096,
        "iops_max" : 177652,
        "iops_mean" : 177159.285714,
        "iops_stddev" : 403.671311,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.328571,
      "sys_cpu" : 49.528571,
      "ctx" : 320288,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.015161,
        "20" : 66.261019,
        "50" : 33.495592,
        "100" : 0.202824,
        "250" : 0.025323,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88647: Tue Oct 25 23:46:32 2022
  read: IOPS=177k, BW=692MiB/s (725MB/s)(4844MiB/7001msec)
    slat (nsec): min=1432, max=25278, avg=2296.35, stdev=1460.01
    clat (usec): min=5, max=315, avg=19.96, stdev= 5.76
     lat (usec): min=10, max=316, avg=22.25, stdev= 5.59
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   16], 10.00th=[   16], 20.00th=[   17],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   20], 60.00th=[   20],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   22], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   77], 99.95th=[   93],
     | 99.99th=[  106]
   bw (  KiB/s): min=704384, max=710608, per=100.00%, avg=708637.86, stdev=1613.81, samples=14
   iops        : min=176096, max=177652, avg=177159.29, stdev=403.67, samples=14
  lat (usec)   : 10=0.02%, 20=66.26%, 50=33.50%, 100=0.20%, 250=0.03%
  lat (usec)   : 500=0.01%
  cpu          : usr=16.33%, sys=49.53%, ctx=320288, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1239993,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=692MiB/s (725MB/s), 692MiB/s-692MiB/s (725MB/s-725MB/s), io=4844MiB (5079MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_4_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_4_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="4" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_4_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_4_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=4
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734406,
  "timestamp_ms" : 1666734406559,
  "time" : "Tue Oct 25 23:46:46 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "4",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 5081395200,
        "io_kbytes" : 4962300,
        "bw_bytes" : 725809912,
        "bw" : 708798,
        "iops" : 177199.114412,
        "runtime" : 7001,
        "total_ios" : 1240571,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1402,
          "max" : 30307,
          "mean" : 2212.269113,
          "stddev" : 1350.794632,
          "N" : 1240572
        },
        "clat_ns" : {
          "min" : 2756,
          "max" : 142718,
          "mean" : 20076.305615,
          "stddev" : 5924.159167,
          "N" : 1240574,
          "percentile" : {
            "1.000000" : 14144,
            "5.000000" : 14784,
            "10.000000" : 15040,
            "20.000000" : 15936,
            "30.000000" : 17280,
            "40.000000" : 17536,
            "50.000000" : 19328,
            "60.000000" : 21376,
            "70.000000" : 21632,
            "80.000000" : 21888,
            "90.000000" : 22144,
            "95.000000" : 31360,
            "99.000000" : 43776,
            "99.500000" : 46336,
            "99.900000" : 80384,
            "99.950000" : 94720,
            "99.990000" : 105984
          }
        },
        "lat_ns" : {
          "min" : 10880,
          "max" : 144201,
          "mean" : 22288.576315,
          "stddev" : 6010.338472,
          "N" : 1240574
        },
        "bw_min" : 700656,
        "bw_max" : 710264,
        "bw_agg" : 100.000000,
        "bw_mean" : 708977.357143,
        "bw_dev" : 2422.535722,
        "bw_samples" : 14,
        "iops_min" : 175164,
        "iops_max" : 177566,
        "iops_mean" : 177244.428571,
        "iops_stddev" : 605.618910,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.228571,
      "sys_cpu" : 50.371429,
      "ctx" : 318778,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.026842,
        "20" : 57.241706,
        "50" : 42.487371,
        "100" : 0.212402,
        "250" : 0.031760,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 4,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=88686: Tue Oct 25 23:46:46 2022
  read: IOPS=177k, BW=692MiB/s (726MB/s)(4846MiB/7001msec)
    slat (nsec): min=1402, max=30307, avg=2212.27, stdev=1350.79
    clat (usec): min=2, max=142, avg=20.08, stdev= 5.92
     lat (usec): min=10, max=144, avg=22.29, stdev= 6.01
    clat percentiles (usec):
     |  1.00th=[   15],  5.00th=[   15], 10.00th=[   16], 20.00th=[   16],
     | 30.00th=[   18], 40.00th=[   18], 50.00th=[   20], 60.00th=[   22],
     | 70.00th=[   22], 80.00th=[   22], 90.00th=[   23], 95.00th=[   32],
     | 99.00th=[   44], 99.50th=[   47], 99.90th=[   81], 99.95th=[   95],
     | 99.99th=[  106]
   bw (  KiB/s): min=700656, max=710264, per=100.00%, avg=708977.36, stdev=2422.54, samples=14
   iops        : min=175164, max=177566, avg=177244.43, stdev=605.62, samples=14
  lat (usec)   : 4=0.01%, 10=0.03%, 20=57.24%, 50=42.49%, 100=0.21%
  lat (usec)   : 250=0.03%
  cpu          : usr=14.23%, sys=50.37%, ctx=318778, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1240571,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=692MiB/s (726MB/s), 692MiB/s-692MiB/s (726MB/s-726MB/s), io=4846MiB (5081MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733129,
  "timestamp_ms" : 1666733129789,
  "time" : "Tue Oct 25 23:25:29 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1804812288,
        "io_kbytes" : 1762512,
        "bw_bytes" : 257793499,
        "bw" : 251751,
        "iops" : 62937.723182,
        "runtime" : 7001,
        "total_ios" : 440627,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2425,
          "max" : 42410,
          "mean" : 2632.123117,
          "stddev" : 277.348007,
          "N" : 440628
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 185869,
          "mean" : 12588.789237,
          "stddev" : 2063.603963,
          "N" : 440627,
          "percentile" : {
            "1.000000" : 12096,
            "5.000000" : 12224,
            "10.000000" : 12224,
            "20.000000" : 12224,
            "30.000000" : 12224,
            "40.000000" : 12352,
            "50.000000" : 12352,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12480,
            "95.000000" : 12608,
            "99.000000" : 22400,
            "99.500000" : 28032,
            "99.900000" : 39168,
            "99.950000" : 50432,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 12183,
          "max" : 188574,
          "mean" : 15220.912393,
          "stddev" : 2084.834946,
          "N" : 440627
        },
        "bw_min" : 250536,
        "bw_max" : 252744,
        "bw_agg" : 100.000000,
        "bw_mean" : 251847.142857,
        "bw_dev" : 664.546445,
        "bw_samples" : 14,
        "iops_min" : 62634,
        "iops_max" : 63186,
        "iops_mean" : 62961.714286,
        "iops_stddev" : 166.188229,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.728571,
      "sys_cpu" : 38.085714,
      "ctx" : 440609,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.589737,
        "50" : 1.345809,
        "100" : 0.053787,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 630140,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 6063,
      "write_ticks" : 0,
      "in_queue" : 6063,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84772: Tue Oct 25 23:25:29 2022
  read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7001msec)
    slat (nsec): min=2425, max=42410, avg=2632.12, stdev=277.35
    clat (nsec): min=210, max=185869, avg=12588.79, stdev=2063.60
     lat (usec): min=12, max=188, avg=15.22, stdev= 2.08
    clat percentiles (nsec):
     |  1.00th=[12096],  5.00th=[12224], 10.00th=[12224], 20.00th=[12224],
     | 30.00th=[12224], 40.00th=[12352], 50.00th=[12352], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
     | 99.00th=[22400], 99.50th=[28032], 99.90th=[39168], 99.95th=[50432],
     | 99.99th=[66048]
   bw (  KiB/s): min=250536, max=252744, per=100.00%, avg=251847.14, stdev=664.55, samples=14
   iops        : min=62634, max=63186, avg=62961.71, stdev=166.19, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.59%, 50=1.35%, 100=0.05%
  lat (usec)   : 250=0.01%
  cpu          : usr=9.73%, sys=38.09%, ctx=440609, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=440627,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1805MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=630140/0, merge=0/0, ticks=6063/0, in_queue=6063, util=99.09%

$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_0.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734420,
  "timestamp_ms" : 1666734420189,
  "time" : "Tue Oct 25 23:47:00 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6420176896,
        "io_kbytes" : 6269704,
        "bw_bytes" : 917037122,
        "bw" : 895544,
        "iops" : 223884.873589,
        "runtime" : 7001,
        "total_ios" : 1567418,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1082,
          "max" : 24687,
          "mean" : 1805.728822,
          "stddev" : 832.327889,
          "N" : 1567419
        },
        "clat_ns" : {
          "min" : 11222,
          "max" : 215775,
          "mean" : 33639.284363,
          "stddev" : 8658.611942,
          "N" : 1567425,
          "percentile" : {
            "1.000000" : 24960,
            "5.000000" : 28800,
            "10.000000" : 29568,
            "20.000000" : 30080,
            "30.000000" : 30080,
            "40.000000" : 30848,
            "50.000000" : 31360,
            "60.000000" : 31616,
            "70.000000" : 31872,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 52480,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 127488,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15389,
          "max" : 218299,
          "mean" : 35445.013616,
          "stddev" : 8573.177387,
          "N" : 1567425
        },
        "bw_min" : 893760,
        "bw_max" : 898008,
        "bw_agg" : 100.000000,
        "bw_mean" : 895765.357143,
        "bw_dev" : 1348.294627,
        "bw_samples" : 14,
        "iops_min" : 223440,
        "iops_max" : 224500,
        "iops_mean" : 223941.285714,
        "iops_stddev" : 336.789920,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.514286,
      "sys_cpu" : 58.171429,
      "ctx" : 756092,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.102462,
        "50" : 91.998178,
        "100" : 7.644419,
        "250" : 0.255388,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88726: Tue Oct 25 23:47:00 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6123MiB/7001msec)
    slat (nsec): min=1082, max=24687, avg=1805.73, stdev=832.33
    clat (usec): min=11, max=215, avg=33.64, stdev= 8.66
     lat (usec): min=15, max=218, avg=35.45, stdev= 8.57
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   30], 20.00th=[   31],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   33], 90.00th=[   46], 95.00th=[   53],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  128], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893760, max=898008, per=100.00%, avg=895765.36, stdev=1348.29, samples=14
   iops        : min=223440, max=224500, avg=223941.29, stdev=336.79, samples=14
  lat (usec)   : 20=0.10%, 50=92.00%, 100=7.64%, 250=0.26%
  cpu          : usr=14.51%, sys=58.17%, ctx=756092, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567418,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6123MiB (6420MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_1.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734433,
  "timestamp_ms" : 1666734433845,
  "time" : "Tue Oct 25 23:47:13 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6416601088,
        "io_kbytes" : 6266212,
        "bw_bytes" : 916526365,
        "bw" : 895045,
        "iops" : 223760.319954,
        "runtime" : 7001,
        "total_ios" : 1566546,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1102,
          "max" : 42740,
          "mean" : 1833.665654,
          "stddev" : 837.174942,
          "N" : 1566546
        },
        "clat_ns" : {
          "min" : 4929,
          "max" : 212439,
          "mean" : 33633.179418,
          "stddev" : 8764.799429,
          "N" : 1566553,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 28800,
            "10.000000" : 29568,
            "20.000000" : 30080,
            "30.000000" : 30080,
            "40.000000" : 30592,
            "50.000000" : 31360,
            "60.000000" : 31616,
            "70.000000" : 31872,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 52480,
            "99.000000" : 62720,
            "99.500000" : 68096,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15479,
          "max" : 215695,
          "mean" : 35466.844208,
          "stddev" : 8680.017687,
          "N" : 1566553
        },
        "bw_min" : 891968,
        "bw_max" : 897104,
        "bw_agg" : 100.000000,
        "bw_mean" : 895253.928571,
        "bw_dev" : 1517.357950,
        "bw_samples" : 14,
        "iops_min" : 222992,
        "iops_max" : 224276,
        "iops_mean" : 223813.428571,
        "iops_stddev" : 379.390141,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.785714,
      "sys_cpu" : 58.928571,
      "ctx" : 753694,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.106285,
        "50" : 92.040195,
        "100" : 7.578456,
        "250" : 0.275447,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88764: Tue Oct 25 23:47:13 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
    slat (nsec): min=1102, max=42740, avg=1833.67, stdev=837.17
    clat (usec): min=4, max=212, avg=33.63, stdev= 8.76
     lat (usec): min=15, max=215, avg=35.47, stdev= 8.68
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   30], 20.00th=[   31],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   33], 90.00th=[   46], 95.00th=[   53],
     | 99.00th=[   63], 99.50th=[   69], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=891968, max=897104, per=100.00%, avg=895253.93, stdev=1517.36, samples=14
   iops        : min=222992, max=224276, avg=223813.43, stdev=379.39, samples=14
  lat (usec)   : 10=0.01%, 20=0.11%, 50=92.04%, 100=7.58%, 250=0.28%
  cpu          : usr=13.79%, sys=58.93%, ctx=753694, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566546,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring_cmd" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-reference_2.txt
io_uring_cmd-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring_cmd, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734447,
  "timestamp_ms" : 1666734447509,
  "time" : "Tue Oct 25 23:47:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring_cmd",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6417027072,
        "io_kbytes" : 6266628,
        "bw_bytes" : 916587212,
        "bw" : 895104,
        "iops" : 223775.032138,
        "runtime" : 7001,
        "total_ios" : 1566649,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1082,
          "max" : 28684,
          "mean" : 1801.328699,
          "stddev" : 837.999667,
          "N" : 1566650
        },
        "clat_ns" : {
          "min" : 11131,
          "max" : 206056,
          "mean" : 33649.558555,
          "stddev" : 8666.871644,
          "N" : 1566656,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 28800,
            "10.000000" : 29824,
            "20.000000" : 30080,
            "30.000000" : 30336,
            "40.000000" : 30848,
            "50.000000" : 31616,
            "60.000000" : 31616,
            "70.000000" : 31872,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 128512,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15359,
          "max" : 208260,
          "mean" : 35450.887665,
          "stddev" : 8587.323029,
          "N" : 1566656
        },
        "bw_min" : 885416,
        "bw_max" : 897224,
        "bw_agg" : 100.000000,
        "bw_mean" : 895327.500000,
        "bw_dev" : 3021.324589,
        "bw_samples" : 14,
        "iops_min" : 221354,
        "iops_max" : 224306,
        "iops_mean" : 223831.857143,
        "iops_stddev" : 755.321415,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 14.185714,
      "sys_cpu" : 58.757143,
      "ctx" : 755492,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.092554,
        "50" : 92.064145,
        "100" : 7.584341,
        "250" : 0.259407,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-reference: (groupid=0, jobs=1): err= 0: pid=88804: Tue Oct 25 23:47:27 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
    slat (nsec): min=1082, max=28684, avg=1801.33, stdev=838.00
    clat (usec): min=11, max=206, avg=33.65, stdev= 8.67
     lat (usec): min=15, max=208, avg=35.45, stdev= 8.59
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   30], 20.00th=[   31],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   33], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  129], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=885416, max=897224, per=100.00%, avg=895327.50, stdev=3021.32, samples=14
   iops        : min=221354, max=224306, avg=223831.86, stdev=755.32, samples=14
  lat (usec)   : 20=0.09%, 50=92.06%, 100=7.58%, 250=0.26%
  cpu          : usr=14.19%, sys=58.76%, ctx=755492, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_0.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734461,
  "timestamp_ms" : 1666734461202,
  "time" : "Tue Oct 25 23:47:41 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6421319680,
        "io_kbytes" : 6270820,
        "bw_bytes" : 917200354,
        "bw" : 895703,
        "iops" : 223924.725039,
        "runtime" : 7001,
        "total_ios" : 1567697,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1152,
          "max" : 175619,
          "mean" : 1747.089774,
          "stddev" : 791.058747,
          "N" : 1567698
        },
        "clat_ns" : {
          "min" : 2465,
          "max" : 215894,
          "mean" : 33673.439684,
          "stddev" : 8679.324358,
          "N" : 1567704,
          "percentile" : {
            "1.000000" : 24960,
            "5.000000" : 28800,
            "10.000000" : 29056,
            "20.000000" : 29824,
            "30.000000" : 30336,
            "40.000000" : 31104,
            "50.000000" : 31616,
            "60.000000" : 31872,
            "70.000000" : 32384,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 66048,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 158720
          }
        },
        "lat_ns" : {
          "min" : 15198,
          "max" : 217137,
          "mean" : 35420.530389,
          "stddev" : 8614.100339,
          "N" : 1567704
        },
        "bw_min" : 894104,
        "bw_max" : 897792,
        "bw_agg" : 100.000000,
        "bw_mean" : 895917.571429,
        "bw_dev" : 1033.548683,
        "bw_samples" : 14,
        "iops_min" : 223526,
        "iops_max" : 224448,
        "iops_mean" : 223979.500000,
        "iops_stddev" : 258.444087,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.142857,
      "sys_cpu" : 56.985714,
      "ctx" : 712649,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.070932,
        "50" : 92.202383,
        "100" : 7.466940,
        "250" : 0.259999,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88843: Tue Oct 25 23:47:41 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
    slat (nsec): min=1152, max=175619, avg=1747.09, stdev=791.06
    clat (usec): min=2, max=215, avg=33.67, stdev= 8.68
     lat (usec): min=15, max=217, avg=35.42, stdev= 8.61
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   30], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  159]
   bw (  KiB/s): min=894104, max=897792, per=100.00%, avg=895917.57, stdev=1033.55, samples=14
   iops        : min=223526, max=224448, avg=223979.50, stdev=258.44, samples=14
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.07%, 50=92.20%, 100=7.47%
  lat (usec)   : 250=0.26%
  cpu          : usr=17.14%, sys=56.99%, ctx=712649, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567697,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_1.txt': No such file or directory

$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_1.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734474,
  "timestamp_ms" : 1666734474814,
  "time" : "Tue Oct 25 23:47:54 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6416293888,
        "io_kbytes" : 6265912,
        "bw_bytes" : 916482486,
        "bw" : 895002,
        "iops" : 223749.607199,
        "runtime" : 7001,
        "total_ios" : 1566471,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1192,
          "max" : 40917,
          "mean" : 1637.510650,
          "stddev" : 539.612945,
          "N" : 1566471
        },
        "clat_ns" : {
          "min" : 221,
          "max" : 212459,
          "mean" : 33807.519823,
          "stddev" : 8996.658594,
          "N" : 1566478,
          "percentile" : {
            "1.000000" : 23936,
            "5.000000" : 27520,
            "10.000000" : 28800,
            "20.000000" : 29568,
            "30.000000" : 30592,
            "40.000000" : 31104,
            "50.000000" : 31616,
            "60.000000" : 32128,
            "70.000000" : 32640,
            "80.000000" : 33536,
            "90.000000" : 45312,
            "95.000000" : 52480,
            "99.000000" : 62720,
            "99.500000" : 68096,
            "99.900000" : 134144,
            "99.950000" : 144384,
            "99.990000" : 160768
          }
        },
        "lat_ns" : {
          "min" : 14056,
          "max" : 213731,
          "mean" : 35445.030097,
          "stddev" : 8951.929801,
          "N" : 1566478
        },
        "bw_min" : 888056,
        "bw_max" : 897410,
        "bw_agg" : 100.000000,
        "bw_mean" : 895350.500000,
        "bw_dev" : 2293.493498,
        "bw_samples" : 14,
        "iops_min" : 222014,
        "iops_max" : 224352,
        "iops_mean" : 223837.571429,
        "iops_stddev" : 573.327228,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.571429,
      "sys_cpu" : 55.157143,
      "ctx" : 696395,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.234029,
        "50" : 91.735755,
        "100" : 7.758267,
        "250" : 0.272204,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88881: Tue Oct 25 23:47:54 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6119MiB/7001msec)
    slat (nsec): min=1192, max=40917, avg=1637.51, stdev=539.61
    clat (nsec): min=221, max=212459, avg=33807.52, stdev=8996.66
     lat (usec): min=14, max=213, avg=35.45, stdev= 8.95
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   28], 10.00th=[   29], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   53],
     | 99.00th=[   63], 99.50th=[   69], 99.90th=[  135], 99.95th=[  145],
     | 99.99th=[  161]
   bw (  KiB/s): min=888056, max=897410, per=100.00%, avg=895350.50, stdev=2293.49, samples=14
   iops        : min=222014, max=224352, avg=223837.57, stdev=573.33, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.23%, 50=91.74%, 100=7.76%
  lat (usec)   : 250=0.27%
  cpu          : usr=17.57%, sys=55.16%, ctx=696395, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566471,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6119MiB (6416MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring_cmd" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/ng0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring_cmd-xnvme_2.txt
io_uring_cmd-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734488,
  "timestamp_ms" : 1666734488432,
  "time" : "Tue Oct 25 23:48:08 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring_cmd",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/ng0n1"
      },
      "read" : {
        "io_bytes" : 6412873728,
        "io_kbytes" : 6262572,
        "bw_bytes" : 915993962,
        "bw" : 894525,
        "iops" : 223630.195686,
        "runtime" : 7001,
        "total_ios" : 1565635,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1132,
          "max" : 29926,
          "mean" : 1678.465054,
          "stddev" : 744.939201,
          "N" : 1565636
        },
        "clat_ns" : {
          "min" : 7093,
          "max" : 395762,
          "mean" : 33756.134564,
          "stddev" : 8803.715628,
          "N" : 1565642,
          "percentile" : {
            "1.000000" : 24960,
            "5.000000" : 28800,
            "10.000000" : 29056,
            "20.000000" : 30080,
            "30.000000" : 30592,
            "40.000000" : 31104,
            "50.000000" : 31616,
            "60.000000" : 31616,
            "70.000000" : 32128,
            "80.000000" : 33024,
            "90.000000" : 45824,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 68096,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15138,
          "max" : 398527,
          "mean" : 35434.600653,
          "stddev" : 8758.561285,
          "N" : 1565642
        },
        "bw_min" : 888520,
        "bw_max" : 898120,
        "bw_agg" : 100.000000,
        "bw_mean" : 894740.571429,
        "bw_dev" : 2660.843987,
        "bw_samples" : 14,
        "iops_min" : 222130,
        "iops_max" : 224530,
        "iops_mean" : 223685.142857,
        "iops_stddev" : 665.288700,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.742857,
      "sys_cpu" : 55.314286,
      "ctx" : 725537,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.082842,
        "50" : 92.137439,
        "100" : 7.499130,
        "250" : 0.280461,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-xnvme: (groupid=0, jobs=1): err= 0: pid=88919: Tue Oct 25 23:48:08 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6116MiB/7001msec)
    slat (nsec): min=1132, max=29926, avg=1678.47, stdev=744.94
    clat (usec): min=7, max=395, avg=33.76, stdev= 8.80
     lat (usec): min=15, max=398, avg=35.43, stdev= 8.76
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   30], 20.00th=[   31],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   69], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=888520, max=898120, per=100.00%, avg=894740.57, stdev=2660.84, samples=14
   iops        : min=222130, max=224530, avg=223685.14, stdev=665.29, samples=14
  lat (usec)   : 10=0.01%, 20=0.08%, 50=92.14%, 100=7.50%, 250=0.28%
  lat (usec)   : 500=0.01%
  cpu          : usr=17.74%, sys=55.31%, ctx=725537, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1565635,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6116MiB (6413MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_0.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734503,
  "timestamp_ms" : 1666734503475,
  "time" : "Tue Oct 25 23:48:23 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6421094400,
        "io_kbytes" : 6270600,
        "bw_bytes" : 917168175,
        "bw" : 895672,
        "iops" : 223917.011855,
        "runtime" : 7001,
        "total_ios" : 1567643,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 661,
          "max" : 38231,
          "mean" : 991.606545,
          "stddev" : 179.577583,
          "N" : 1567643
        },
        "clat_ns" : {
          "min" : 1854,
          "max" : 230002,
          "mean" : 34445.715295,
          "stddev" : 8475.141091,
          "N" : 1567649,
          "percentile" : {
            "1.000000" : 28032,
            "5.000000" : 30080,
            "10.000000" : 30592,
            "20.000000" : 31104,
            "30.000000" : 31360,
            "40.000000" : 31616,
            "50.000000" : 31872,
            "60.000000" : 32384,
            "70.000000" : 32640,
            "80.000000" : 33536,
            "90.000000" : 44288,
            "95.000000" : 52480,
            "99.000000" : 62720,
            "99.500000" : 66048,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 13204,
          "max" : 230923,
          "mean" : 35437.321713,
          "stddev" : 8475.876232,
          "N" : 1567649
        },
        "bw_min" : 893891,
        "bw_max" : 897088,
        "bw_agg" : 100.000000,
        "bw_mean" : 895900.214286,
        "bw_dev" : 990.139709,
        "bw_samples" : 14,
        "iops_min" : 223472,
        "iops_max" : 224272,
        "iops_mean" : 223975.000000,
        "iops_stddev" : 247.652051,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 0.720955,
        "8" : 99.261694,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.995006,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.022008,
        "50" : 91.171332,
        "100" : 8.551245,
        "250" : 0.255160,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=88970: Tue Oct 25 23:48:23 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
    slat (nsec): min=661, max=38231, avg=991.61, stdev=179.58
    clat (nsec): min=1854, max=230002, avg=34445.72, stdev=8475.14
     lat (usec): min=13, max=230, avg=35.44, stdev= 8.48
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   31], 10.00th=[   31], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   45], 95.00th=[   53],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893891, max=897088, per=100.00%, avg=895900.21, stdev=990.14, samples=14
   iops        : min=223472, max=224272, avg=223975.00, stdev=247.65, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.02%, 50=91.17%
  lat (usec)   : 100=8.55%, 250=0.26%
  cpu          : usr=99.99%, sys=0.00%, ctx=18, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.7%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567643,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_1.txt"
$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_1.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734518,
  "timestamp_ms" : 1666734518563,
  "time" : "Tue Oct 25 23:48:38 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6416740352,
        "io_kbytes" : 6266348,
        "bw_bytes" : 916546257,
        "bw" : 895064,
        "iops" : 223765.033567,
        "runtime" : 7001,
        "total_ios" : 1566579,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 651,
          "max" : 10229,
          "mean" : 922.635981,
          "stddev" : 74.486330,
          "N" : 1566580
        },
        "clat_ns" : {
          "min" : 13977,
          "max" : 229741,
          "mean" : 34515.006953,
          "stddev" : 8893.514425,
          "N" : 1566582,
          "percentile" : {
            "1.000000" : 24192,
            "5.000000" : 30080,
            "10.000000" : 30336,
            "20.000000" : 30592,
            "30.000000" : 31104,
            "40.000000" : 31872,
            "50.000000" : 32384,
            "60.000000" : 32640,
            "70.000000" : 33024,
            "80.000000" : 33536,
            "90.000000" : 45312,
            "95.000000" : 52480,
            "99.000000" : 63744,
            "99.500000" : 69120,
            "99.900000" : 136192,
            "99.950000" : 144384,
            "99.990000" : 160768
          }
        },
        "lat_ns" : {
          "min" : 14748,
          "max" : 230652,
          "mean" : 35437.642907,
          "stddev" : 8893.112545,
          "N" : 1566582
        },
        "bw_min" : 892728,
        "bw_max" : 898024,
        "bw_agg" : 100.000000,
        "bw_mean" : 895313.500000,
        "bw_dev" : 1341.825436,
        "bw_samples" : 14,
        "iops_min" : 223182,
        "iops_max" : 224506,
        "iops_mean" : 223828.357143,
        "iops_stddev" : 335.467895,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.100000,
        "4" : 0.602140,
        "8" : 99.388540,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.995614,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.048769,
        "50" : 91.253298,
        "100" : 8.411577,
        "250" : 0.286548,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89020: Tue Oct 25 23:48:38 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
    slat (nsec): min=651, max=10229, avg=922.64, stdev=74.49
    clat (usec): min=13, max=229, avg=34.52, stdev= 8.89
     lat (usec): min=14, max=230, avg=35.44, stdev= 8.89
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   34], 80.00th=[   34], 90.00th=[   46], 95.00th=[   53],
     | 99.00th=[   64], 99.50th=[   70], 99.90th=[  137], 99.95th=[  145],
     | 99.99th=[  161]
   bw (  KiB/s): min=892728, max=898024, per=100.00%, avg=895313.50, stdev=1341.83, samples=14
   iops        : min=223182, max=224506, avg=223828.36, stdev=335.47, samples=14
  lat (usec)   : 20=0.05%, 50=91.25%, 100=8.41%, 250=0.29%
  cpu          : usr=99.99%, sys=0.00%, ctx=4, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.1%, 4=0.6%, 8=99.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566579,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring_cmd-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="0" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring_cmd.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring_cmd-bdev_xnvme_2.txt
io_uring_cmd-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734533,
  "timestamp_ms" : 1666734533650,
  "time" : "Tue Oct 25 23:48:53 2022",
  "jobs" : [
    {
      "jobname" : "io_uring_cmd-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring_cmd-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "0",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring_cmd.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419181568,
        "io_kbytes" : 6268732,
        "bw_bytes" : 916894953,
        "bw" : 895405,
        "iops" : 223850.307099,
        "runtime" : 7001,
        "total_ios" : 1567176,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 651,
          "max" : 7023,
          "mean" : 994.223934,
          "stddev" : 136.149078,
          "N" : 1567176
        },
        "clat_ns" : {
          "min" : 11421,
          "max" : 178565,
          "mean" : 34501.146111,
          "stddev" : 8517.845352,
          "N" : 1567183,
          "percentile" : {
            "1.000000" : 28544,
            "5.000000" : 30080,
            "10.000000" : 30592,
            "20.000000" : 31104,
            "30.000000" : 31616,
            "40.000000" : 31872,
            "50.000000" : 31872,
            "60.000000" : 32384,
            "70.000000" : 32640,
            "80.000000" : 33536,
            "90.000000" : 44288,
            "95.000000" : 52480,
            "99.000000" : 63232,
            "99.500000" : 66048,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 154624
          }
        },
        "lat_ns" : {
          "min" : 12323,
          "max" : 179727,
          "mean" : 35495.369933,
          "stddev" : 8518.963941,
          "N" : 1567183
        },
        "bw_min" : 892608,
        "bw_max" : 897816,
        "bw_agg" : 100.000000,
        "bw_mean" : 895630.214286,
        "bw_dev" : 1342.108684,
        "bw_samples" : 14,
        "iops_min" : 223152,
        "iops_max" : 224454,
        "iops_mean" : 223907.500000,
        "iops_stddev" : 335.521351,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 99.985714,
      "sys_cpu" : 0.000000,
      "ctx" : 4,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.100000,
        "4" : 0.630050,
        "8" : 99.368801,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.997888,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.013847,
        "50" : 91.109358,
        "100" : 8.605415,
        "250" : 0.271827,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring_cmd-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89070: Tue Oct 25 23:48:53 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=651, max=7023, avg=994.22, stdev=136.15
    clat (usec): min=11, max=178, avg=34.50, stdev= 8.52
     lat (usec): min=12, max=179, avg=35.50, stdev= 8.52
    clat percentiles (usec):
     |  1.00th=[   29],  5.00th=[   31], 10.00th=[   31], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   45], 95.00th=[   53],
     | 99.00th=[   64], 99.50th=[   67], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  155]
   bw (  KiB/s): min=892608, max=897816, per=100.00%, avg=895630.21, stdev=1342.11, samples=14
   iops        : min=223152, max=224454, avg=223907.50, stdev=335.52, samples=14
  lat (usec)   : 20=0.01%, 50=91.11%, 100=8.61%, 250=0.27%
  cpu          : usr=99.99%, sys=0.00%, ctx=4, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.1%, 4=0.6%, 8=99.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567176,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring-reference_0.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734547,
  "timestamp_ms" : 1666734547398,
  "time" : "Tue Oct 25 23:49:07 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418763776,
        "io_kbytes" : 6268324,
        "bw_bytes" : 916835277,
        "bw" : 895346,
        "iops" : 223835.594915,
        "runtime" : 7001,
        "total_ios" : 1567073,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1222,
          "max" : 43772,
          "mean" : 1732.981963,
          "stddev" : 738.776513,
          "N" : 1567074
        },
        "clat_ns" : {
          "min" : 80,
          "max" : 210004,
          "mean" : 33654.755580,
          "stddev" : 8683.484113,
          "N" : 1567080,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 28544,
            "10.000000" : 28800,
            "20.000000" : 29568,
            "30.000000" : 30592,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31872,
            "70.000000" : 32384,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 66048,
            "99.900000" : 128512,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15339,
          "max" : 211277,
          "mean" : 35387.737334,
          "stddev" : 8632.811278,
          "N" : 1567080
        },
        "bw_min" : 892488,
        "bw_max" : 898184,
        "bw_agg" : 100.000000,
        "bw_mean" : 895707.214286,
        "bw_dev" : 1505.889371,
        "bw_samples" : 14,
        "iops_min" : 223122,
        "iops_max" : 224546,
        "iops_mean" : 223926.928571,
        "iops_stddev" : 376.422525,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.885714,
      "sys_cpu" : 58.414286,
      "ctx" : 669792,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.010000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.103697,
        "50" : 92.161693,
        "100" : 7.477826,
        "250" : 0.257040,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2235843,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 70932,
      "write_ticks" : 0,
      "in_queue" : 70932,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89109: Tue Oct 25 23:49:07 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
    slat (nsec): min=1222, max=43772, avg=1732.98, stdev=738.78
    clat (nsec): min=80, max=210004, avg=33654.76, stdev=8683.48
     lat (usec): min=15, max=211, avg=35.39, stdev= 8.63
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   29], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  129], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892488, max=898184, per=100.00%, avg=895707.21, stdev=1505.89, samples=14
   iops        : min=223122, max=224546, avg=223926.93, stdev=376.42, samples=14
  lat (nsec)   : 100=0.01%
  lat (usec)   : 10=0.01%, 20=0.10%, 50=92.16%, 100=7.48%, 250=0.26%
  cpu          : usr=16.89%, sys=58.41%, ctx=669792, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567073,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2235843/0, merge=0/0, ticks=70932/0, in_queue=70932, util=99.09%

$ rm /tmp/fio-output_4K_8_io_uring-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733143,
  "timestamp_ms" : 1666733143402,
  "time" : "Tue Oct 25 23:25:43 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1824374784,
        "io_kbytes" : 1781616,
        "bw_bytes" : 260587742,
        "bw" : 254480,
        "iops" : 63620.054278,
        "runtime" : 7001,
        "total_ios" : 445404,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2424,
          "max" : 34645,
          "mean" : 2592.826618,
          "stddev" : 222.766188,
          "N" : 445404
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 122219,
          "mean" : 12494.563899,
          "stddev" : 2015.777434,
          "N" : 445404,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 11968,
            "10.000000" : 11968,
            "20.000000" : 12096,
            "30.000000" : 12096,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12480,
            "95.000000" : 12480,
            "99.000000" : 22400,
            "99.500000" : 27776,
            "99.900000" : 33024,
            "99.950000" : 49920,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 12434,
          "max" : 124784,
          "mean" : 15087.390517,
          "stddev" : 2032.769095,
          "N" : 445404
        },
        "bw_min" : 253272,
        "bw_max" : 255968,
        "bw_agg" : 100.000000,
        "bw_mean" : 254578.357143,
        "bw_dev" : 701.625103,
        "bw_samples" : 14,
        "iops_min" : 63318,
        "iops_max" : 63992,
        "iops_mean" : 63644.571429,
        "iops_stddev" : 175.405778,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.200000,
      "sys_cpu" : 36.728571,
      "ctx" : 445388,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.600821,
        "50" : 1.339009,
        "100" : 0.052537,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 638282,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 6018,
      "write_ticks" : 0,
      "in_queue" : 6018,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84810: Tue Oct 25 23:25:43 2022
  read: IOPS=63.6k, BW=249MiB/s (261MB/s)(1740MiB/7001msec)
    slat (nsec): min=2424, max=34645, avg=2592.83, stdev=222.77
    clat (nsec): min=240, max=122219, avg=12494.56, stdev=2015.78
     lat (usec): min=12, max=124, avg=15.09, stdev= 2.03
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[11968], 10.00th=[11968], 20.00th=[12096],
     | 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12480],
     | 99.00th=[22400], 99.50th=[27776], 99.90th=[33024], 99.95th=[49920],
     | 99.99th=[64768]
   bw (  KiB/s): min=253272, max=255968, per=100.00%, avg=254578.36, stdev=701.63, samples=14
   iops        : min=63318, max=63992, avg=63644.57, stdev=175.41, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=0.01%, 20=98.60%, 50=1.34%, 100=0.05%, 250=0.01%
  cpu          : usr=11.20%, sys=36.73%, ctx=445388, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=445404,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=249MiB/s (261MB/s), 249MiB/s-249MiB/s (261MB/s-261MB/s), io=1740MiB (1824MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=638282/0, merge=0/0, ticks=6018/0, in_queue=6018, util=99.09%

$ cat /tmp/fio-output_4K_8_io_uring-reference_1.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734561,
  "timestamp_ms" : 1666734561035,
  "time" : "Tue Oct 25 23:49:21 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418571264,
        "io_kbytes" : 6268136,
        "bw_bytes" : 916807779,
        "bw" : 895320,
        "iops" : 223828.881588,
        "runtime" : 7001,
        "total_ios" : 1567026,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1212,
          "max" : 36078,
          "mean" : 1714.383513,
          "stddev" : 722.179076,
          "N" : 1567027
        },
        "clat_ns" : {
          "min" : 12804,
          "max" : 206107,
          "mean" : 33675.974473,
          "stddev" : 8691.726209,
          "N" : 1567033,
          "percentile" : {
            "1.000000" : 24960,
            "5.000000" : 28544,
            "10.000000" : 28800,
            "20.000000" : 29824,
            "30.000000" : 30592,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31616,
            "70.000000" : 32384,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 128512,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15399,
          "max" : 207379,
          "mean" : 35390.359061,
          "stddev" : 8641.981671,
          "N" : 1567033
        },
        "bw_min" : 893378,
        "bw_max" : 897136,
        "bw_agg" : 100.000000,
        "bw_mean" : 895682.857143,
        "bw_dev" : 1185.473801,
        "bw_samples" : 14,
        "iops_min" : 223344,
        "iops_max" : 224284,
        "iops_mean" : 223920.642857,
        "iops_stddev" : 296.455888,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.600000,
      "sys_cpu" : 58.057143,
      "ctx" : 693034,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.086406,
        "50" : 92.191195,
        "100" : 7.458842,
        "250" : 0.264003,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2235997,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 71144,
      "write_ticks" : 0,
      "in_queue" : 71144,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89147: Tue Oct 25 23:49:21 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
    slat (nsec): min=1212, max=36078, avg=1714.38, stdev=722.18
    clat (usec): min=12, max=206, avg=33.68, stdev= 8.69
     lat (usec): min=15, max=207, avg=35.39, stdev= 8.64
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   29], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  129], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893378, max=897136, per=100.00%, avg=895682.86, stdev=1185.47, samples=14
   iops        : min=223344, max=224284, avg=223920.64, stdev=296.46, samples=14
  lat (usec)   : 20=0.09%, 50=92.19%, 100=7.46%, 250=0.26%
  cpu          : usr=16.60%, sys=58.06%, ctx=693034, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567026,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2235997/0, merge=0/0, ticks=71144/0, in_queue=71144, util=99.09%

$ rm /tmp/fio-output_4K_8_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-reference_2.txt"
$ cat /tmp/fio-output_4K_8_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734574,
  "timestamp_ms" : 1666734574665,
  "time" : "Tue Oct 25 23:49:34 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419468288,
        "io_kbytes" : 6269012,
        "bw_bytes" : 916935907,
        "bw" : 895445,
        "iops" : 223860.162834,
        "runtime" : 7001,
        "total_ios" : 1567245,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1232,
          "max" : 53460,
          "mean" : 1707.207037,
          "stddev" : 717.038812,
          "N" : 1567246
        },
        "clat_ns" : {
          "min" : 7434,
          "max" : 196628,
          "mean" : 33672.910071,
          "stddev" : 8730.705101,
          "N" : 1567252,
          "percentile" : {
            "1.000000" : 25216,
            "5.000000" : 28544,
            "10.000000" : 28800,
            "20.000000" : 29312,
            "30.000000" : 30592,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31872,
            "70.000000" : 32384,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15579,
          "max" : 198051,
          "mean" : 35380.117355,
          "stddev" : 8681.842147,
          "N" : 1567252
        },
        "bw_min" : 892529,
        "bw_max" : 897192,
        "bw_agg" : 100.000000,
        "bw_mean" : 895805.071429,
        "bw_dev" : 1222.038050,
        "bw_samples" : 14,
        "iops_min" : 223132,
        "iops_max" : 224298,
        "iops_mean" : 223951.357143,
        "iops_stddev" : 305.643233,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 17.300000,
      "sys_cpu" : 57.928571,
      "ctx" : 667803,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.066550,
        "50" : 92.209706,
        "100" : 7.450845,
        "250" : 0.273154,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2236081,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 71107,
      "write_ticks" : 0,
      "in_queue" : 71107,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=89185: Tue Oct 25 23:49:34 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=1232, max=53460, avg=1707.21, stdev=717.04
    clat (usec): min=7, max=196, avg=33.67, stdev= 8.73
     lat (usec): min=15, max=198, avg=35.38, stdev= 8.68
    clat percentiles (usec):
     |  1.00th=[   26],  5.00th=[   29], 10.00th=[   29], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892529, max=897192, per=100.00%, avg=895805.07, stdev=1222.04, samples=14
   iops        : min=223132, max=224298, avg=223951.36, stdev=305.64, samples=14
  lat (usec)   : 10=0.01%, 20=0.07%, 50=92.21%, 100=7.45%, 250=0.27%
  cpu          : usr=17.30%, sys=57.93%, ctx=667803, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567245,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2236081/0, merge=0/0, ticks=71107/0, in_queue=71107, util=99.09%

$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734589,
  "timestamp_ms" : 1666734589681,
  "time" : "Tue Oct 25 23:49:49 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419386368,
        "io_kbytes" : 6268932,
        "bw_bytes" : 916924206,
        "bw" : 895433,
        "iops" : 223857.448936,
        "runtime" : 7001,
        "total_ios" : 1567226,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 17783,
          "mean" : 177.147500,
          "stddev" : 78.228730,
          "N" : 1567226
        },
        "clat_ns" : {
          "min" : 13115,
          "max" : 1356474,
          "mean" : 35300.071299,
          "stddev" : 8917.372350,
          "N" : 1567233,
          "percentile" : {
            "1.000000" : 29568,
            "5.000000" : 32384,
            "10.000000" : 32384,
            "20.000000" : 32640,
            "30.000000" : 32640,
            "40.000000" : 32640,
            "50.000000" : 32640,
            "60.000000" : 32640,
            "70.000000" : 33024,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 53504,
            "99.000000" : 63744,
            "99.500000" : 67072,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 13395,
          "max" : 1356644,
          "mean" : 35477.218807,
          "stddev" : 8916.741132,
          "N" : 1567233
        },
        "bw_min" : 893360,
        "bw_max" : 897640,
        "bw_agg" : 100.000000,
        "bw_mean" : 895644.571429,
        "bw_dev" : 1100.192830,
        "bw_samples" : 14,
        "iops_min" : 223340,
        "iops_max" : 224410,
        "iops_mean" : 223911.142857,
        "iops_stddev" : 275.048208,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.585714,
      "sys_cpu" : 60.385714,
      "ctx" : 40,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.291773,
        "8" : 98.704335,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.996949,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.010000,
        "50" : 90.676265,
        "100" : 9.055108,
        "250" : 0.262374,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89233: Tue Oct 25 23:49:49 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=110, max=17783, avg=177.15, stdev=78.23
    clat (usec): min=13, max=1356, avg=35.30, stdev= 8.92
     lat (usec): min=13, max=1356, avg=35.48, stdev= 8.92
    clat percentiles (usec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   33], 20.00th=[   33],
     | 30.00th=[   33], 40.00th=[   33], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   34], 80.00th=[   34], 90.00th=[   46], 95.00th=[   54],
     | 99.00th=[   64], 99.50th=[   68], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893360, max=897640, per=100.00%, avg=895644.57, stdev=1100.19, samples=14
   iops        : min=223340, max=224410, avg=223911.14, stdev=275.05, samples=14
  lat (usec)   : 20=0.01%, 50=90.68%, 100=9.06%, 250=0.26%
  lat (msec)   : 2=0.01%
  cpu          : usr=39.59%, sys=60.39%, ctx=40, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567226,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734605,
  "timestamp_ms" : 1666734605226,
  "time" : "Tue Oct 25 23:50:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418595840,
        "io_kbytes" : 6268160,
        "bw_bytes" : 916811289,
        "bw" : 895323,
        "iops" : 223829.881446,
        "runtime" : 7001,
        "total_ios" : 1567033,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 120,
          "max" : 18484,
          "mean" : 180.659000,
          "stddev" : 80.296904,
          "N" : 1567033
        },
        "clat_ns" : {
          "min" : 13105,
          "max" : 191649,
          "mean" : 35285.179822,
          "stddev" : 8502.394509,
          "N" : 1567040,
          "percentile" : {
            "1.000000" : 29824,
            "5.000000" : 32384,
            "10.000000" : 32384,
            "20.000000" : 32640,
            "30.000000" : 32640,
            "40.000000" : 32640,
            "50.000000" : 32640,
            "60.000000" : 32640,
            "70.000000" : 33024,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 53504,
            "99.000000" : 63744,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 13355,
          "max" : 191830,
          "mean" : 35465.838813,
          "stddev" : 8501.391034,
          "N" : 1567040
        },
        "bw_min" : 891984,
        "bw_max" : 898280,
        "bw_agg" : 100.000000,
        "bw_mean" : 895534.142857,
        "bw_dev" : 1537.441324,
        "bw_samples" : 14,
        "iops_min" : 222996,
        "iops_max" : 224570,
        "iops_mean" : 223883.500000,
        "iops_stddev" : 384.365761,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 42.957143,
      "sys_cpu" : 57.014286,
      "ctx" : 11,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.244645,
        "8" : 98.750441,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.996629,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.010000,
        "50" : 90.670713,
        "100" : 9.049012,
        "250" : 0.275425,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89282: Tue Oct 25 23:50:05 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
    slat (nsec): min=120, max=18484, avg=180.66, stdev=80.30
    clat (usec): min=13, max=191, avg=35.29, stdev= 8.50
     lat (usec): min=13, max=191, avg=35.47, stdev= 8.50
    clat percentiles (usec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   33], 20.00th=[   33],
     | 30.00th=[   33], 40.00th=[   33], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   34], 80.00th=[   34], 90.00th=[   46], 95.00th=[   54],
     | 99.00th=[   64], 99.50th=[   68], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=891984, max=898280, per=100.00%, avg=895534.14, stdev=1537.44, samples=14
   iops        : min=222996, max=224570, avg=223883.50, stdev=384.37, samples=14
  lat (usec)   : 20=0.01%, 50=90.67%, 100=9.05%, 250=0.28%
  cpu          : usr=42.96%, sys=57.01%, ctx=11, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.2%, 8=98.8%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567033,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-reference_2.txt': No such file or directory

$ rm /tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734620,
  "timestamp_ms" : 1666734620429,
  "time" : "Tue Oct 25 23:50:20 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6415040512,
        "io_kbytes" : 6264688,
        "bw_bytes" : 916303458,
        "bw" : 894827,
        "iops" : 223705.899157,
        "runtime" : 7001,
        "total_ios" : 1566165,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 110,
          "max" : 41308,
          "mean" : 199.295783,
          "stddev" : 128.404646,
          "N" : 1566165
        },
        "clat_ns" : {
          "min" : 12804,
          "max" : 177943,
          "mean" : 35310.666389,
          "stddev" : 8531.519283,
          "N" : 1566172,
          "percentile" : {
            "1.000000" : 29568,
            "5.000000" : 32384,
            "10.000000" : 32384,
            "20.000000" : 32640,
            "30.000000" : 32640,
            "40.000000" : 32640,
            "50.000000" : 32640,
            "60.000000" : 32640,
            "70.000000" : 33024,
            "80.000000" : 33024,
            "90.000000" : 45312,
            "95.000000" : 53504,
            "99.000000" : 63744,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 142336,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 13055,
          "max" : 178154,
          "mean" : 35509.962140,
          "stddev" : 8530.220824,
          "N" : 1566172
        },
        "bw_min" : 893232,
        "bw_max" : 895976,
        "bw_agg" : 100.000000,
        "bw_mean" : 895036.428571,
        "bw_dev" : 819.653277,
        "bw_samples" : 14,
        "iops_min" : 223308,
        "iops_max" : 223994,
        "iops_mean" : 223759.214286,
        "iops_stddev" : 205.000630,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 39.642857,
      "sys_cpu" : 60.328571,
      "ctx" : 19,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.353178,
        "8" : 98.618983,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.991192,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.011110,
        "50" : 90.645749,
        "100" : 9.069670,
        "250" : 0.273917,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=89331: Tue Oct 25 23:50:20 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6118MiB/7001msec)
    slat (nsec): min=110, max=41308, avg=199.30, stdev=128.40
    clat (usec): min=12, max=177, avg=35.31, stdev= 8.53
     lat (usec): min=13, max=178, avg=35.51, stdev= 8.53
    clat percentiles (usec):
     |  1.00th=[   30],  5.00th=[   33], 10.00th=[   33], 20.00th=[   33],
     | 30.00th=[   33], 40.00th=[   33], 50.00th=[   33], 60.00th=[   33],
     | 70.00th=[   34], 80.00th=[   34], 90.00th=[   46], 95.00th=[   54],
     | 99.00th=[   64], 99.50th=[   68], 99.90th=[  133], 99.95th=[  143],
     | 99.99th=[  157]
   bw (  KiB/s): min=893232, max=895976, per=100.00%, avg=895036.43, stdev=819.65, samples=14
   iops        : min=223308, max=223994, avg=223759.21, stdev=205.00, samples=14
  lat (usec)   : 20=0.01%, 50=90.65%, 100=9.07%, 250=0.27%
  cpu          : usr=39.64%, sys=60.33%, ctx=19, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.4%, 8=98.6%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566165,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6118MiB (6415MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734635,
  "timestamp_ms" : 1666734635553,
  "time" : "Tue Oct 25 23:50:35 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6416384000,
        "io_kbytes" : 6266000,
        "bw_bytes" : 916495357,
        "bw" : 895014,
        "iops" : 223752.749607,
        "runtime" : 7001,
        "total_ios" : 1566493,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1482,
          "max" : 67687,
          "mean" : 1754.159928,
          "stddev" : 246.058703,
          "N" : 1566493
        },
        "clat_ns" : {
          "min" : 15098,
          "max" : 202079,
          "mean" : 33741.697037,
          "stddev" : 8649.734313,
          "N" : 1566500,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 30848,
            "10.000000" : 30848,
            "20.000000" : 30848,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31104,
            "60.000000" : 31104,
            "70.000000" : 31360,
            "80.000000" : 31360,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 17352,
          "max" : 203803,
          "mean" : 35495.856783,
          "stddev" : 8639.780490,
          "N" : 1566500
        },
        "bw_min" : 894056,
        "bw_max" : 897080,
        "bw_agg" : 100.000000,
        "bw_mean" : 895219.785714,
        "bw_dev" : 929.395517,
        "bw_samples" : 14,
        "iops_min" : 223514,
        "iops_max" : 224270,
        "iops_mean" : 223804.785714,
        "iops_stddev" : 232.359198,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 30.385714,
      "sys_cpu" : 69.585714,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.026684,
        "50" : 91.094183,
        "100" : 8.585867,
        "250" : 0.293713,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89380: Tue Oct 25 23:50:35 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6119MiB/7001msec)
    slat (nsec): min=1482, max=67687, avg=1754.16, stdev=246.06
    clat (usec): min=15, max=202, avg=33.74, stdev= 8.65
     lat (usec): min=17, max=203, avg=35.50, stdev= 8.64
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=894056, max=897080, per=100.00%, avg=895219.79, stdev=929.40, samples=14
   iops        : min=223514, max=224270, avg=223804.79, stdev=232.36, samples=14
  lat (usec)   : 20=0.03%, 50=91.09%, 100=8.59%, 250=0.29%
  cpu          : usr=30.39%, sys=69.59%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566493,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6119MiB (6416MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734650,
  "timestamp_ms" : 1666734650706,
  "time" : "Tue Oct 25 23:50:50 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418513920,
        "io_kbytes" : 6268080,
        "bw_bytes" : 916799588,
        "bw" : 895312,
        "iops" : 223827.024711,
        "runtime" : 7001,
        "total_ios" : 1567013,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1473,
          "max" : 25368,
          "mean" : 1749.800455,
          "stddev" : 234.769642,
          "N" : 1567013
        },
        "clat_ns" : {
          "min" : 12604,
          "max" : 1083192,
          "mean" : 33745.384779,
          "stddev" : 8890.164459,
          "N" : 1567020,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 30848,
            "10.000000" : 30848,
            "20.000000" : 30848,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31104,
            "60.000000" : 31104,
            "70.000000" : 31360,
            "80.000000" : 31360,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 66048,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 14317,
          "max" : 1084925,
          "mean" : 35495.185114,
          "stddev" : 8880.807151,
          "N" : 1567020
        },
        "bw_min" : 893456,
        "bw_max" : 896816,
        "bw_agg" : 100.000000,
        "bw_mean" : 895527.357143,
        "bw_dev" : 964.990044,
        "bw_samples" : 14,
        "iops_min" : 223364,
        "iops_max" : 224204,
        "iops_mean" : 223881.785714,
        "iops_stddev" : 241.228962,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 25.757143,
      "sys_cpu" : 74.214286,
      "ctx" : 42,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.046075,
        "50" : 91.074356,
        "100" : 8.607331,
        "250" : 0.272174,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89429: Tue Oct 25 23:50:50 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
    slat (nsec): min=1473, max=25368, avg=1749.80, stdev=234.77
    clat (usec): min=12, max=1083, avg=33.75, stdev= 8.89
     lat (usec): min=14, max=1084, avg=35.50, stdev= 8.88
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893456, max=896816, per=100.00%, avg=895527.36, stdev=964.99, samples=14
   iops        : min=223364, max=224204, avg=223881.79, stdev=241.23, samples=14
  lat (usec)   : 20=0.05%, 50=91.07%, 100=8.61%, 250=0.27%
  lat (msec)   : 2=0.01%
  cpu          : usr=25.76%, sys=74.21%, ctx=42, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567013,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="io_uring" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-reference_2.txt"
$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734665,
  "timestamp_ms" : 1666734665909,
  "time" : "Tue Oct 25 23:51:05 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418640896,
        "io_kbytes" : 6268204,
        "bw_bytes" : 916817725,
        "bw" : 895329,
        "iops" : 223831.452650,
        "runtime" : 7001,
        "total_ios" : 1567044,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1472,
          "max" : 25949,
          "mean" : 1754.159593,
          "stddev" : 224.643680,
          "N" : 1567044
        },
        "clat_ns" : {
          "min" : 15079,
          "max" : 220213,
          "mean" : 33737.245896,
          "stddev" : 8603.637014,
          "N" : 1567051,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 30848,
            "10.000000" : 30848,
            "20.000000" : 30848,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31104,
            "60.000000" : 31104,
            "70.000000" : 31360,
            "80.000000" : 31360,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 66048,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 17322,
          "max" : 221956,
          "mean" : 35491.405287,
          "stddev" : 8595.079541,
          "N" : 1567051
        },
        "bw_min" : 892688,
        "bw_max" : 897683,
        "bw_agg" : 100.000000,
        "bw_mean" : 895548.785714,
        "bw_dev" : 1287.687264,
        "bw_samples" : 14,
        "iops_min" : 223172,
        "iops_max" : 224420,
        "iops_mean" : 223887.142857,
        "iops_stddev" : 321.708625,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 29.585714,
      "sys_cpu" : 70.385714,
      "ctx" : 13,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.035098,
        "50" : 91.088827,
        "100" : 8.597908,
        "250" : 0.278614,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89479: Tue Oct 25 23:51:05 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6121MiB/7001msec)
    slat (nsec): min=1472, max=25949, avg=1754.16, stdev=224.64
    clat (usec): min=15, max=220, avg=33.74, stdev= 8.60
     lat (usec): min=17, max=221, avg=35.49, stdev= 8.60
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892688, max=897683, per=100.00%, avg=895548.79, stdev=1287.69, samples=14
   iops        : min=223172, max=224420, avg=223887.14, stdev=321.71, samples=14
  lat (usec)   : 20=0.04%, 50=91.09%, 100=8.60%, 250=0.28%
  cpu          : usr=29.59%, sys=70.39%, ctx=13, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567044,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6121MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_8_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734679,
  "timestamp_ms" : 1666734679679,
  "time" : "Tue Oct 25 23:51:19 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6417080320,
        "io_kbytes" : 6266680,
        "bw_bytes" : 916594817,
        "bw" : 895112,
        "iops" : 223777.031853,
        "runtime" : 7001,
        "total_ios" : 1566663,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1312,
          "max" : 178064,
          "mean" : 2022.793881,
          "stddev" : 860.278382,
          "N" : 1566663
        },
        "clat_ns" : {
          "min" : 5811,
          "max" : 218590,
          "mean" : 33429.997546,
          "stddev" : 8920.942316,
          "N" : 1566670,
          "percentile" : {
            "1.000000" : 23936,
            "5.000000" : 28032,
            "10.000000" : 28032,
            "20.000000" : 28544,
            "30.000000" : 30336,
            "40.000000" : 30848,
            "50.000000" : 31872,
            "60.000000" : 32128,
            "70.000000" : 32384,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15940,
          "max" : 221325,
          "mean" : 35452.790505,
          "stddev" : 8811.650507,
          "N" : 1566670
        },
        "bw_min" : 894184,
        "bw_max" : 897432,
        "bw_agg" : 100.000000,
        "bw_mean" : 895345.000000,
        "bw_dev" : 952.534675,
        "bw_samples" : 14,
        "iops_min" : 223546,
        "iops_max" : 224358,
        "iops_mean" : 223836.214286,
        "iops_stddev" : 238.146748,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 16.414286,
      "sys_cpu" : 62.785714,
      "ctx" : 539835,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.228958,
        "50" : 92.130024,
        "100" : 7.357294,
        "250" : 0.284107,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89518: Tue Oct 25 23:51:19 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
    slat (nsec): min=1312, max=178064, avg=2022.79, stdev=860.28
    clat (usec): min=5, max=218, avg=33.43, stdev= 8.92
     lat (usec): min=15, max=221, avg=35.45, stdev= 8.81
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   29], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   33], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=894184, max=897432, per=100.00%, avg=895345.00, stdev=952.53, samples=14
   iops        : min=223546, max=224358, avg=223836.21, stdev=238.15, samples=14
  lat (usec)   : 10=0.01%, 20=0.23%, 50=92.13%, 100=7.36%, 250=0.28%
  cpu          : usr=16.41%, sys=62.79%, ctx=539835, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566663,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734693,
  "timestamp_ms" : 1666734693305,
  "time" : "Tue Oct 25 23:51:33 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6416519168,
        "io_kbytes" : 6266132,
        "bw_bytes" : 916514664,
        "bw" : 895033,
        "iops" : 223757.320383,
        "runtime" : 7001,
        "total_ios" : 1566525,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1283,
          "max" : 37260,
          "mean" : 2104.457766,
          "stddev" : 885.717940,
          "N" : 1566526
        },
        "clat_ns" : {
          "min" : 220,
          "max" : 267692,
          "mean" : 33332.316245,
          "stddev" : 8926.514508,
          "N" : 1566532,
          "percentile" : {
            "1.000000" : 24192,
            "5.000000" : 28032,
            "10.000000" : 28032,
            "20.000000" : 28288,
            "30.000000" : 29568,
            "40.000000" : 30848,
            "50.000000" : 31872,
            "60.000000" : 32128,
            "70.000000" : 32384,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15189,
          "max" : 269315,
          "mean" : 35436.773715,
          "stddev" : 8807.724468,
          "N" : 1566532
        },
        "bw_min" : 892808,
        "bw_max" : 897400,
        "bw_agg" : 100.000000,
        "bw_mean" : 895258.142857,
        "bw_dev" : 1198.668098,
        "bw_samples" : 14,
        "iops_min" : 223202,
        "iops_max" : 224350,
        "iops_mean" : 223814.500000,
        "iops_stddev" : 299.664107,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.414286,
      "sys_cpu" : 64.100000,
      "ctx" : 529590,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.256364,
        "50" : 92.233191,
        "100" : 7.231356,
        "250" : 0.279089,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89557: Tue Oct 25 23:51:33 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6119MiB/7001msec)
    slat (nsec): min=1283, max=37260, avg=2104.46, stdev=885.72
    clat (nsec): min=220, max=267692, avg=33332.32, stdev=8926.51
     lat (usec): min=15, max=269, avg=35.44, stdev= 8.81
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   33], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892808, max=897400, per=100.00%, avg=895258.14, stdev=1198.67, samples=14
   iops        : min=223202, max=224350, avg=223814.50, stdev=299.66, samples=14
  lat (nsec)   : 250=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.26%, 50=92.23%, 100=7.23%
  lat (usec)   : 250=0.28%, 500=0.01%
  cpu          : usr=15.41%, sys=64.10%, ctx=529590, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566525,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6119MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring-reference_2.txt
io_uring-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=io_uring, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733157,
  "timestamp_ms" : 1666733157028,
  "time" : "Tue Oct 25 23:25:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "io_uring",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1804107776,
        "io_kbytes" : 1761824,
        "bw_bytes" : 257729682,
        "bw" : 251689,
        "iops" : 62922.285714,
        "runtime" : 7000,
        "total_ios" : 440456,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2465,
          "max" : 34736,
          "mean" : 2646.838867,
          "stddev" : 213.387163,
          "N" : 440456
        },
        "clat_ns" : {
          "min" : 240,
          "max" : 105898,
          "mean" : 12567.897549,
          "stddev" : 1992.233414,
          "N" : 440456,
          "percentile" : {
            "1.000000" : 12096,
            "5.000000" : 12224,
            "10.000000" : 12224,
            "20.000000" : 12224,
            "30.000000" : 12224,
            "40.000000" : 12224,
            "50.000000" : 12352,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12352,
            "95.000000" : 12608,
            "99.000000" : 22400,
            "99.500000" : 27520,
            "99.900000" : 36608,
            "99.950000" : 49920,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 11952,
          "max" : 108593,
          "mean" : 15214.736416,
          "stddev" : 2006.232859,
          "N" : 440456
        },
        "bw_min" : 250789,
        "bw_max" : 252440,
        "bw_agg" : 100.000000,
        "bw_mean" : 251748.785714,
        "bw_dev" : 511.742450,
        "bw_samples" : 14,
        "iops_min" : 62697,
        "iops_max" : 63110,
        "iops_mean" : 62937.142857,
        "iops_stddev" : 127.995106,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.485714,
      "sys_cpu" : 38.314286,
      "ctx" : 440441,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.659571,
        "50" : 1.284351,
        "100" : 0.047905,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 623550,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5990,
      "write_ticks" : 0,
      "in_queue" : 5990,
      "util" : 99.088748
    }
  ]
}

io_uring-reference: (groupid=0, jobs=1): err= 0: pid=84848: Tue Oct 25 23:25:57 2022
  read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7000msec)
    slat (nsec): min=2465, max=34736, avg=2646.84, stdev=213.39
    clat (nsec): min=240, max=105898, avg=12567.90, stdev=1992.23
     lat (usec): min=11, max=108, avg=15.21, stdev= 2.01
    clat percentiles (nsec):
     |  1.00th=[12096],  5.00th=[12224], 10.00th=[12224], 20.00th=[12224],
     | 30.00th=[12224], 40.00th=[12224], 50.00th=[12352], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
     | 99.00th=[22400], 99.50th=[27520], 99.90th=[36608], 99.95th=[49920],
     | 99.99th=[67072]
   bw (  KiB/s): min=250789, max=252440, per=100.00%, avg=251748.79, stdev=511.74, samples=14
   iops        : min=62697, max=63110, avg=62937.14, stdev=128.00, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.66%, 50=1.28%, 100=0.05%
  lat (usec)   : 250=0.01%
  cpu          : usr=9.49%, sys=38.31%, ctx=440441, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=440456,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1804MB), run=7000-7000msec

Disk stats (read/write):
  nvme0n1: ios=623550/0, merge=0/0, ticks=5990/0, in_queue=5990, util=99.09%

$ cat /tmp/fio-output_4K_8_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734706,
  "timestamp_ms" : 1666734706937,
  "time" : "Tue Oct 25 23:51:46 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6417715200,
        "io_kbytes" : 6267300,
        "bw_bytes" : 916685502,
        "bw" : 895200,
        "iops" : 223799.028710,
        "runtime" : 7001,
        "total_ios" : 1566817,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1292,
          "max" : 57117,
          "mean" : 2005.485569,
          "stddev" : 861.647587,
          "N" : 1566818
        },
        "clat_ns" : {
          "min" : 261,
          "max" : 206247,
          "mean" : 33462.330400,
          "stddev" : 8878.873955,
          "N" : 1566824,
          "percentile" : {
            "1.000000" : 23936,
            "5.000000" : 28032,
            "10.000000" : 28032,
            "20.000000" : 28544,
            "30.000000" : 30336,
            "40.000000" : 30848,
            "50.000000" : 31872,
            "60.000000" : 32128,
            "70.000000" : 32384,
            "80.000000" : 32640,
            "90.000000" : 45312,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 16311,
          "max" : 207599,
          "mean" : 35467.815561,
          "stddev" : 8771.771322,
          "N" : 1566824
        },
        "bw_min" : 893368,
        "bw_max" : 897939,
        "bw_agg" : 100.000000,
        "bw_mean" : 895407.071429,
        "bw_dev" : 1490.748416,
        "bw_samples" : 14,
        "iops_min" : 223342,
        "iops_max" : 224484,
        "iops_mean" : 223851.714286,
        "iops_stddev" : 372.323149,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 15.357143,
      "sys_cpu" : 63.742857,
      "ctx" : 541721,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.272080,
        "50" : 92.033020,
        "100" : 7.417778,
        "250" : 0.277314,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=89597: Tue Oct 25 23:51:46 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
    slat (nsec): min=1292, max=57117, avg=2005.49, stdev=861.65
    clat (nsec): min=261, max=206247, avg=33462.33, stdev=8878.87
     lat (usec): min=16, max=207, avg=35.47, stdev= 8.77
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   29], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   33], 90.00th=[   46], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893368, max=897939, per=100.00%, avg=895407.07, stdev=1490.75, samples=14
   iops        : min=223342, max=224484, avg=223851.71, stdev=372.32, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.27%, 50=92.03%, 100=7.42%
  lat (usec)   : 250=0.28%
  cpu          : usr=15.36%, sys=63.74%, ctx=541721, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566817,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6418MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734720,
  "timestamp_ms" : 1666734720577,
  "time" : "Tue Oct 25 23:52:00 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6421184512,
        "io_kbytes" : 6270688,
        "bw_bytes" : 917181047,
        "bw" : 895684,
        "iops" : 223920.011427,
        "runtime" : 7001,
        "total_ios" : 1567664,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1372,
          "max" : 23945,
          "mean" : 2044.074586,
          "stddev" : 778.052273,
          "N" : 1567665
        },
        "clat_ns" : {
          "min" : 15299,
          "max" : 193182,
          "mean" : 33405.325188,
          "stddev" : 8836.893542,
          "N" : 1567671,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 28288,
            "10.000000" : 28288,
            "20.000000" : 28800,
            "30.000000" : 29312,
            "40.000000" : 30592,
            "50.000000" : 31616,
            "60.000000" : 32128,
            "70.000000" : 32384,
            "80.000000" : 33536,
            "90.000000" : 44800,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 128512,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 17783,
          "max" : 194615,
          "mean" : 35449.399103,
          "stddev" : 8754.063693,
          "N" : 1567671
        },
        "bw_min" : 894608,
        "bw_max" : 897456,
        "bw_agg" : 100.000000,
        "bw_mean" : 896045.714286,
        "bw_dev" : 800.433289,
        "bw_samples" : 14,
        "iops_min" : 223652,
        "iops_max" : 224364,
        "iops_mean" : 224011.428571,
        "iops_stddev" : 200.229758,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 21.657143,
      "sys_cpu" : 57.442857,
      "ctx" : 533446,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.044142,
        "50" : 92.606005,
        "100" : 7.090231,
        "250" : 0.260068,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2235295,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 71869,
      "write_ticks" : 0,
      "in_queue" : 71869,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=89636: Tue Oct 25 23:52:00 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
    slat (nsec): min=1372, max=23945, avg=2044.07, stdev=778.05
    clat (usec): min=15, max=193, avg=33.41, stdev= 8.84
     lat (usec): min=17, max=194, avg=35.45, stdev= 8.75
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   45], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  129], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=894608, max=897456, per=100.00%, avg=896045.71, stdev=800.43, samples=14
   iops        : min=223652, max=224364, avg=224011.43, stdev=200.23, samples=14
  lat (usec)   : 20=0.04%, 50=92.61%, 100=7.09%, 250=0.26%
  cpu          : usr=21.66%, sys=57.44%, ctx=533446, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567664,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6421MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2235295/0, merge=0/0, ticks=71869/0, in_queue=71869, util=99.09%

$ rm /tmp/fio-output_4K_8_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734734,
  "timestamp_ms" : 1666734734194,
  "time" : "Tue Oct 25 23:52:14 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419886080,
        "io_kbytes" : 6269420,
        "bw_bytes" : 916995583,
        "bw" : 895503,
        "iops" : 223874.875018,
        "runtime" : 7001,
        "total_ios" : 1567348,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1422,
          "max" : 42911,
          "mean" : 2070.390944,
          "stddev" : 758.523721,
          "N" : 1567348
        },
        "clat_ns" : {
          "min" : 391,
          "max" : 177383,
          "mean" : 33384.698160,
          "stddev" : 8925.881402,
          "N" : 1567355,
          "percentile" : {
            "1.000000" : 22912,
            "5.000000" : 27264,
            "10.000000" : 28544,
            "20.000000" : 28800,
            "30.000000" : 29056,
            "40.000000" : 29568,
            "50.000000" : 30848,
            "60.000000" : 32128,
            "70.000000" : 32640,
            "80.000000" : 34048,
            "90.000000" : 44800,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 128512,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 14407,
          "max" : 178875,
          "mean" : 35455.088327,
          "stddev" : 8862.894253,
          "N" : 1567355
        },
        "bw_min" : 893552,
        "bw_max" : 897856,
        "bw_agg" : 100.000000,
        "bw_mean" : 895919.923077,
        "bw_dev" : 1175.135344,
        "bw_samples" : 13,
        "iops_min" : 223388,
        "iops_max" : 224464,
        "iops_mean" : 223979.923077,
        "iops_stddev" : 293.956477,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 25.357143,
      "sys_cpu" : 52.514286,
      "ctx" : 563635,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.000000,
        "20" : 0.058570,
        "50" : 92.628185,
        "100" : 7.051848,
        "250" : 0.260950,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2177732,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 69367,
      "write_ticks" : 0,
      "in_queue" : 69367,
      "util" : 99.085923
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=89676: Tue Oct 25 23:52:14 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=1422, max=42911, avg=2070.39, stdev=758.52
    clat (nsec): min=391, max=177383, avg=33384.70, stdev=8925.88
     lat (usec): min=14, max=178, avg=35.46, stdev= 8.86
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   28], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   30], 50.00th=[   31], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   35], 90.00th=[   45], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  129], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893552, max=897856, per=100.00%, avg=895919.92, stdev=1175.14, samples=13
   iops        : min=223388, max=224464, avg=223979.92, stdev=293.96, samples=13
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 20=0.06%, 50=92.63%, 100=7.05%, 250=0.26%
  cpu          : usr=25.36%, sys=52.51%, ctx=563635, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567348,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6122MiB (6420MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2177732/0, merge=0/0, ticks=69367/0, in_queue=69367, util=99.09%

$ rm /tmp/fio-output_4K_8_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_8_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734747,
  "timestamp_ms" : 1666734747818,
  "time" : "Tue Oct 25 23:52:27 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419091456,
        "io_kbytes" : 6268644,
        "bw_bytes" : 916882081,
        "bw" : 895392,
        "iops" : 223847.164691,
        "runtime" : 7001,
        "total_ios" : 1567154,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1332,
          "max" : 1297263,
          "mean" : 2019.222772,
          "stddev" : 1292.972535,
          "N" : 1567154
        },
        "clat_ns" : {
          "min" : 12954,
          "max" : 1329383,
          "mean" : 33446.079906,
          "stddev" : 9235.556673,
          "N" : 1567161,
          "percentile" : {
            "1.000000" : 24192,
            "5.000000" : 28032,
            "10.000000" : 28288,
            "20.000000" : 28800,
            "30.000000" : 29568,
            "40.000000" : 30592,
            "50.000000" : 31616,
            "60.000000" : 32128,
            "70.000000" : 32384,
            "80.000000" : 33536,
            "90.000000" : 44800,
            "95.000000" : 51968,
            "99.000000" : 62720,
            "99.500000" : 67072,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 17462,
          "max" : 1332139,
          "mean" : 35465.301490,
          "stddev" : 9217.281644,
          "N" : 1567161
        },
        "bw_min" : 892584,
        "bw_max" : 897699,
        "bw_agg" : 100.000000,
        "bw_mean" : 895743.142857,
        "bw_dev" : 1372.428272,
        "bw_samples" : 14,
        "iops_min" : 223146,
        "iops_max" : 224424,
        "iops_mean" : 223935.714286,
        "iops_stddev" : 343.066584,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.885714,
      "sys_cpu" : 55.914286,
      "ctx" : 540455,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999936,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.049261,
        "50" : 92.557081,
        "100" : 7.134079,
        "250" : 0.259579,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 2234710,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 71868,
      "write_ticks" : 0,
      "in_queue" : 71868,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=89716: Tue Oct 25 23:52:27 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=1332, max=1297.3k, avg=2019.22, stdev=1292.97
    clat (usec): min=12, max=1329, avg=33.45, stdev= 9.24
     lat (usec): min=17, max=1332, avg=35.47, stdev= 9.22
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   29], 10.00th=[   29], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   33], 80.00th=[   34], 90.00th=[   45], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892584, max=897699, per=100.00%, avg=895743.14, stdev=1372.43, samples=14
   iops        : min=223146, max=224424, avg=223935.71, stdev=343.07, samples=14
  lat (usec)   : 20=0.05%, 50=92.56%, 100=7.13%, 250=0.26%
  lat (msec)   : 2=0.01%
  cpu          : usr=22.89%, sys=55.91%, ctx=540455, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567154,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=2234710/0, merge=0/0, ticks=71868/0, in_queue=71868, util=99.09%

$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt': No such file or directory

$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734762,
  "timestamp_ms" : 1666734762792,
  "time" : "Tue Oct 25 23:52:42 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6415536128,
        "io_kbytes" : 6265172,
        "bw_bytes" : 916374250,
        "bw" : 894896,
        "iops" : 223723.039566,
        "runtime" : 7001,
        "total_ios" : 1566285,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1553,
          "max" : 184837,
          "mean" : 1667.843817,
          "stddev" : 320.534687,
          "N" : 1566286
        },
        "clat_ns" : {
          "min" : 541,
          "max" : 215374,
          "mean" : 33789.432111,
          "stddev" : 8671.015722,
          "N" : 1566285,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 30848,
            "10.000000" : 30848,
            "20.000000" : 31104,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31360,
            "70.000000" : 31360,
            "80.000000" : 31360,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 66048,
            "99.900000" : 134144,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 14217,
          "max" : 217007,
          "mean" : 35457.275931,
          "stddev" : 8658.022173,
          "N" : 1566285
        },
        "bw_min" : 893960,
        "bw_max" : 896200,
        "bw_agg" : 100.000000,
        "bw_mean" : 895105.071429,
        "bw_dev" : 806.860627,
        "bw_samples" : 14,
        "iops_min" : 223490,
        "iops_max" : 224050,
        "iops_mean" : 223776.214286,
        "iops_stddev" : 201.682225,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 67.585714,
      "sys_cpu" : 32.385714,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.306531,
        "8" : 98.689702,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.996755,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.047437,
        "50" : 90.986251,
        "100" : 8.674794,
        "250" : 0.291326,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89765: Tue Oct 25 23:52:42 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6118MiB/7001msec)
    slat (nsec): min=1553, max=184837, avg=1667.84, stdev=320.53
    clat (nsec): min=541, max=215374, avg=33789.43, stdev=8671.02
     lat (usec): min=14, max=217, avg=35.46, stdev= 8.66
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  135], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893960, max=896200, per=100.00%, avg=895105.07, stdev=806.86, samples=14
   iops        : min=223490, max=224050, avg=223776.21, stdev=201.68, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.05%, 50=90.99%, 100=8.67%
  lat (usec)   : 250=0.29%
  cpu          : usr=67.59%, sys=32.39%, ctx=14, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566285,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6118MiB (6416MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734777,
  "timestamp_ms" : 1666734777907,
  "time" : "Tue Oct 25 23:52:57 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419636224,
        "io_kbytes" : 6269176,
        "bw_bytes" : 916959894,
        "bw" : 895468,
        "iops" : 223866.161977,
        "runtime" : 7001,
        "total_ios" : 1567287,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1562,
          "max" : 23695,
          "mean" : 1655.664724,
          "stddev" : 232.107596,
          "N" : 1567287
        },
        "clat_ns" : {
          "min" : 7424,
          "max" : 174908,
          "mean" : 33773.507443,
          "stddev" : 8548.878283,
          "N" : 1567294,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 30848,
            "10.000000" : 31104,
            "20.000000" : 31104,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31360,
            "70.000000" : 31360,
            "80.000000" : 31360,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 66048,
            "99.900000" : 130560,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 14487,
          "max" : 176551,
          "mean" : 35429.171972,
          "stddev" : 8534.978594,
          "N" : 1567294
        },
        "bw_min" : 892608,
        "bw_max" : 898528,
        "bw_agg" : 100.000000,
        "bw_mean" : 895684.785714,
        "bw_dev" : 1449.683476,
        "bw_samples" : 14,
        "iops_min" : 223152,
        "iops_max" : 224632,
        "iops_mean" : 223921.142857,
        "iops_stddev" : 362.336617,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 68.328571,
      "sys_cpu" : 31.642857,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.275835,
        "8" : 98.722952,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.997849,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.037517,
        "50" : 91.020981,
        "100" : 8.667525,
        "250" : 0.274359,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89814: Tue Oct 25 23:52:57 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=1562, max=23695, avg=1655.66, stdev=232.11
    clat (usec): min=7, max=174, avg=33.77, stdev= 8.55
     lat (usec): min=14, max=176, avg=35.43, stdev= 8.53
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   32], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  131], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=892608, max=898528, per=100.00%, avg=895684.79, stdev=1449.68, samples=14
   iops        : min=223152, max=224632, avg=223921.14, stdev=362.34, samples=14
  lat (usec)   : 10=0.01%, 20=0.04%, 50=91.02%, 100=8.67%, 250=0.27%
  cpu          : usr=68.33%, sys=31.64%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567287,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6420MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734793,
  "timestamp_ms" : 1666734793038,
  "time" : "Tue Oct 25 23:53:13 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6418944000,
        "io_kbytes" : 6268500,
        "bw_bytes" : 916861019,
        "bw" : 895372,
        "iops" : 223841.879731,
        "runtime" : 7001,
        "total_ios" : 1567117,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1542,
          "max" : 51917,
          "mean" : 1638.791215,
          "stddev" : 252.351930,
          "N" : 1567118
        },
        "clat_ns" : {
          "min" : 12954,
          "max" : 173766,
          "mean" : 33817.441936,
          "stddev" : 8542.652012,
          "N" : 1567117,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 30848,
            "10.000000" : 31104,
            "20.000000" : 31104,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31360,
            "70.000000" : 31360,
            "80.000000" : 31616,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 66048,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15129,
          "max" : 175399,
          "mean" : 35456.233161,
          "stddev" : 8527.216217,
          "N" : 1567117
        },
        "bw_min" : 894312,
        "bw_max" : 896672,
        "bw_agg" : 100.000000,
        "bw_mean" : 895574.142857,
        "bw_dev" : 761.594467,
        "bw_samples" : 14,
        "iops_min" : 223578,
        "iops_max" : 224168,
        "iops_mean" : 223893.500000,
        "iops_stddev" : 190.409579,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 71.014286,
      "sys_cpu" : 28.971429,
      "ctx" : 46,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.324087,
        "8" : 98.671446,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.995953,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.057941,
        "50" : 90.970872,
        "100" : 8.694820,
        "250" : 0.276367,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=89863: Tue Oct 25 23:53:13 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6122MiB/7001msec)
    slat (nsec): min=1542, max=51917, avg=1638.79, stdev=252.35
    clat (usec): min=12, max=173, avg=33.82, stdev= 8.54
     lat (usec): min=15, max=175, avg=35.46, stdev= 8.53
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   32], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   67], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=894312, max=896672, per=100.00%, avg=895574.14, stdev=761.59, samples=14
   iops        : min=223578, max=224168, avg=223893.50, stdev=190.41, samples=14
  lat (usec)   : 20=0.06%, 50=90.97%, 100=8.69%, 250=0.28%
  cpu          : usr=71.01%, sys=28.97%, ctx=46, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.3%, 8=98.7%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567117,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6122MiB (6419MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734808,
  "timestamp_ms" : 1666734808232,
  "time" : "Tue Oct 25 23:53:28 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6424309760,
        "io_kbytes" : 6273740,
        "bw_bytes" : 917627447,
        "bw" : 896120,
        "iops" : 224028.995858,
        "runtime" : 7001,
        "total_ios" : 1568427,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1683,
          "max" : 37610,
          "mean" : 1920.989116,
          "stddev" : 147.325480,
          "N" : 1568428
        },
        "clat_ns" : {
          "min" : 1113,
          "max" : 185858,
          "mean" : 33505.838697,
          "stddev" : 8743.561843,
          "N" : 1568430,
          "percentile" : {
            "1.000000" : 23168,
            "5.000000" : 28288,
            "10.000000" : 30336,
            "20.000000" : 30592,
            "30.000000" : 30848,
            "40.000000" : 30848,
            "50.000000" : 30848,
            "60.000000" : 31104,
            "70.000000" : 31104,
            "80.000000" : 31360,
            "90.000000" : 44800,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 67072,
            "99.900000" : 130560,
            "99.950000" : 142336,
            "99.990000" : 160768
          }
        },
        "lat_ns" : {
          "min" : 13776,
          "max" : 187792,
          "mean" : 35426.827932,
          "stddev" : 8743.318794,
          "N" : 1568430
        },
        "bw_min" : 894136,
        "bw_max" : 898096,
        "bw_agg" : 100.000000,
        "bw_mean" : 896613.785714,
        "bw_dev" : 1166.083001,
        "bw_samples" : 14,
        "iops_min" : 223534,
        "iops_max" : 224524,
        "iops_mean" : 224153.357143,
        "iops_stddev" : 291.452492,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 41.528571,
      "sys_cpu" : 58.442857,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 0.739403,
        "8" : 99.259832,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999232,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.127963,
        "50" : 91.673441,
        "100" : 7.954977,
        "250" : 0.243747,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89912: Tue Oct 25 23:53:28 2022
  read: IOPS=224k, BW=875MiB/s (918MB/s)(6127MiB/7001msec)
    slat (nsec): min=1683, max=37610, avg=1920.99, stdev=147.33
    clat (nsec): min=1113, max=185858, avg=33505.84, stdev=8743.56
     lat (usec): min=13, max=187, avg=35.43, stdev= 8.74
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   29], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   31], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   45], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  131], 99.95th=[  143],
     | 99.99th=[  161]
   bw (  KiB/s): min=894136, max=898096, per=100.00%, avg=896613.79, stdev=1166.08, samples=14
   iops        : min=223534, max=224524, avg=224153.36, stdev=291.45, samples=14
  lat (usec)   : 2=0.01%, 20=0.13%, 50=91.67%, 100=7.95%, 250=0.24%
  cpu          : usr=41.53%, sys=58.44%, ctx=18, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=0.7%, 8=99.3%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1568427,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (918MB/s), 875MiB/s-875MiB/s (918MB/s-918MB/s), io=6127MiB (6424MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734823,
  "timestamp_ms" : 1666734823417,
  "time" : "Tue Oct 25 23:53:43 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6421917696,
        "io_kbytes" : 6271404,
        "bw_bytes" : 917285772,
        "bw" : 895786,
        "iops" : 223945.722040,
        "runtime" : 7001,
        "total_ios" : 1567844,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1623,
          "max" : 74419,
          "mean" : 1755.833047,
          "stddev" : 331.760129,
          "N" : 1567844
        },
        "clat_ns" : {
          "min" : 1042,
          "max" : 177283,
          "mean" : 33637.009474,
          "stddev" : 8459.789321,
          "N" : 1567851,
          "percentile" : {
            "1.000000" : 24704,
            "5.000000" : 30592,
            "10.000000" : 30848,
            "20.000000" : 30848,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31104,
            "60.000000" : 31360,
            "70.000000" : 31360,
            "80.000000" : 31616,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 64768,
            "99.900000" : 127488,
            "99.950000" : 138240,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 13115,
          "max" : 178976,
          "mean" : 35392.842113,
          "stddev" : 8437.240273,
          "N" : 1567851
        },
        "bw_min" : 894312,
        "bw_max" : 897768,
        "bw_agg" : 100.000000,
        "bw_mean" : 896021.142857,
        "bw_dev" : 1070.104442,
        "bw_samples" : 14,
        "iops_min" : 223578,
        "iops_max" : 224442,
        "iops_mean" : 224005.285714,
        "iops_stddev" : 267.558312,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 48.500000,
      "sys_cpu" : 51.471429,
      "ctx" : 25,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.606537,
        "8" : 98.385299,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.995820,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.055618,
        "50" : 91.209266,
        "100" : 8.477757,
        "250" : 0.257360,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=89963: Tue Oct 25 23:53:43 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6124MiB/7001msec)
    slat (nsec): min=1623, max=74419, avg=1755.83, stdev=331.76
    clat (nsec): min=1042, max=177283, avg=33637.01, stdev=8459.79
     lat (usec): min=13, max=178, avg=35.39, stdev= 8.44
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   31],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   65], 99.90th=[  128], 99.95th=[  139],
     | 99.99th=[  157]
   bw (  KiB/s): min=894312, max=897768, per=100.00%, avg=896021.14, stdev=1070.10, samples=14
   iops        : min=223578, max=224442, avg=224005.29, stdev=267.56, samples=14
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.06%, 50=91.21%, 100=8.48%
  lat (usec)   : 250=0.26%
  cpu          : usr=48.50%, sys=51.47%, ctx=25, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.6%, 8=98.4%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6124MiB (6422MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_8_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734838,
  "timestamp_ms" : 1666734838539,
  "time" : "Tue Oct 25 23:53:58 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 6419918848,
        "io_kbytes" : 6269452,
        "bw_bytes" : 917000263,
        "bw" : 895508,
        "iops" : 223876.017712,
        "runtime" : 7001,
        "total_ios" : 1567356,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1583,
          "max" : 180989,
          "mean" : 1700.893440,
          "stddev" : 355.436322,
          "N" : 1567356
        },
        "clat_ns" : {
          "min" : 1022,
          "max" : 221516,
          "mean" : 33718.714563,
          "stddev" : 8534.695940,
          "N" : 1567363,
          "percentile" : {
            "1.000000" : 24448,
            "5.000000" : 30592,
            "10.000000" : 30848,
            "20.000000" : 31104,
            "30.000000" : 31104,
            "40.000000" : 31104,
            "50.000000" : 31104,
            "60.000000" : 31360,
            "70.000000" : 31360,
            "80.000000" : 31616,
            "90.000000" : 43776,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 65280,
            "99.900000" : 129536,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 15730,
          "max" : 223139,
          "mean" : 35419.607731,
          "stddev" : 8515.962916,
          "N" : 1567363
        },
        "bw_min" : 894160,
        "bw_max" : 898496,
        "bw_agg" : 100.000000,
        "bw_mean" : 895716.928571,
        "bw_dev" : 996.908025,
        "bw_samples" : 14,
        "iops_min" : 223540,
        "iops_max" : 224624,
        "iops_mean" : 223929.357143,
        "iops_stddev" : 249.118324,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 48.785714,
      "sys_cpu" : 51.185714,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.100000,
        "2" : 0.100000,
        "4" : 1.516120,
        "8" : 98.479159,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.995950,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 0.054040,
        "50" : 91.150256,
        "100" : 8.527929,
        "250" : 0.267967,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=90012: Tue Oct 25 23:53:58 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6123MiB/7001msec)
    slat (nsec): min=1583, max=180989, avg=1700.89, stdev=355.44
    clat (nsec): min=1022, max=221516, avg=33718.71, stdev=8534.70
     lat (usec): min=15, max=223, avg=35.42, stdev= 8.52
    clat percentiles (usec):
     |  1.00th=[   25],  5.00th=[   31], 10.00th=[   31], 20.00th=[   32],
     | 30.00th=[   32], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   32], 80.00th=[   32], 90.00th=[   44], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   66], 99.90th=[  130], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=894160, max=898496, per=100.00%, avg=895716.93, stdev=996.91, samples=14
   iops        : min=223540, max=224624, avg=223929.36, stdev=249.12, samples=14
  lat (usec)   : 2=0.01%, 10=0.01%, 20=0.05%, 50=91.15%, 100=8.53%
  lat (usec)   : 250=0.27%
  cpu          : usr=48.79%, sys=51.19%, ctx=10, majf=0, minf=0
  IO depths    : 1=0.1%, 2=0.1%, 4=1.5%, 8=98.5%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1567356,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6123MiB (6420MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_0.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733172,
  "timestamp_ms" : 1666733172034,
  "time" : "Tue Oct 25 23:26:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2409836544,
        "io_kbytes" : 2353356,
        "bw_bytes" : 344213190,
        "bw" : 336145,
        "iops" : 84036.423368,
        "runtime" : 7001,
        "total_ios" : 588339,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 10810,
          "mean" : 173.542789,
          "stddev" : 22.695499,
          "N" : 588339
        },
        "clat_ns" : {
          "min" : 10410,
          "max" : 78336,
          "mean" : 11484.418524,
          "stddev" : 2190.358012,
          "N" : 588339,
          "percentile" : {
            "1.000000" : 10944,
            "5.000000" : 10944,
            "10.000000" : 11072,
            "20.000000" : 11072,
            "30.000000" : 11072,
            "40.000000" : 11072,
            "50.000000" : 11072,
            "60.000000" : 11200,
            "70.000000" : 11200,
            "80.000000" : 11328,
            "90.000000" : 11328,
            "95.000000" : 11968,
            "99.000000" : 22144,
            "99.500000" : 28288,
            "99.900000" : 28544,
            "99.950000" : 43776,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 10580,
          "max" : 78517,
          "mean" : 11657.961313,
          "stddev" : 2190.707867,
          "N" : 588339
        },
        "bw_min" : 335000,
        "bw_max" : 337450,
        "bw_agg" : 100.000000,
        "bw_mean" : 336330.692308,
        "bw_dev" : 777.678960,
        "bw_samples" : 13,
        "iops_min" : 83750,
        "iops_max" : 84362,
        "iops_mean" : 84082.615385,
        "iops_stddev" : 194.345714,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 83.014286,
      "sys_cpu" : 16.971429,
      "ctx" : 16,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 98.617124,
        "50" : 1.344293,
        "100" : 0.038583,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84897: Tue Oct 25 23:26:12 2022
  read: IOPS=84.0k, BW=328MiB/s (344MB/s)(2298MiB/7001msec)
    slat (nsec): min=160, max=10810, avg=173.54, stdev=22.70
    clat (nsec): min=10410, max=78336, avg=11484.42, stdev=2190.36
     lat (nsec): min=10580, max=78517, avg=11657.96, stdev=2190.71
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[10944], 10.00th=[11072], 20.00th=[11072],
     | 30.00th=[11072], 40.00th=[11072], 50.00th=[11072], 60.00th=[11200],
     | 70.00th=[11200], 80.00th=[11328], 90.00th=[11328], 95.00th=[11968],
     | 99.00th=[22144], 99.50th=[28288], 99.90th=[28544], 99.95th=[43776],
     | 99.99th=[68096]
   bw (  KiB/s): min=335000, max=337450, per=100.00%, avg=336330.69, stdev=777.68, samples=13
   iops        : min=83750, max=84362, avg=84082.62, stdev=194.35, samples=13
  lat (usec)   : 20=98.62%, 50=1.34%, 100=0.04%
  cpu          : usr=83.01%, sys=16.97%, ctx=16, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=588339,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=328MiB/s (344MB/s), 328MiB/s-328MiB/s (344MB/s-344MB/s), io=2298MiB (2410MB), run=7001-7001msec

$ cat /tmp/fio-output_4K_8_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734852,
  "timestamp_ms" : 1666734852329,
  "time" : "Tue Oct 25 23:54:12 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6413660160,
        "io_kbytes" : 6263340,
        "bw_bytes" : 916106293,
        "bw" : 894635,
        "iops" : 223657.620340,
        "runtime" : 7001,
        "total_ios" : 1565827,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1463,
          "max" : 42490,
          "mean" : 1879.747679,
          "stddev" : 476.780147,
          "N" : 1565828
        },
        "clat_ns" : {
          "min" : 2766,
          "max" : 250209,
          "mean" : 33625.378656,
          "stddev" : 9412.417229,
          "N" : 1565834,
          "percentile" : {
            "1.000000" : 22912,
            "5.000000" : 25984,
            "10.000000" : 27264,
            "20.000000" : 28544,
            "30.000000" : 29312,
            "40.000000" : 30336,
            "50.000000" : 31104,
            "60.000000" : 31872,
            "70.000000" : 33024,
            "80.000000" : 36608,
            "90.000000" : 45312,
            "95.000000" : 52480,
            "99.000000" : 63232,
            "99.500000" : 69120,
            "99.900000" : 134144,
            "99.950000" : 142336,
            "99.990000" : 160768
          }
        },
        "lat_ns" : {
          "min" : 15689,
          "max" : 262933,
          "mean" : 35505.126049,
          "stddev" : 9396.777144,
          "N" : 1565834
        },
        "bw_min" : 887864,
        "bw_max" : 897378,
        "bw_agg" : 100.000000,
        "bw_mean" : 895042.928571,
        "bw_dev" : 2300.390016,
        "bw_samples" : 14,
        "iops_min" : 221966,
        "iops_max" : 224344,
        "iops_mean" : 223760.500000,
        "iops_stddev" : 574.996020,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 22.085714,
      "sys_cpu" : 54.371429,
      "ctx" : 539751,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.000000,
        "20" : 0.414669,
        "50" : 92.437734,
        "100" : 6.875664,
        "250" : 0.272188,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90051: Tue Oct 25 23:54:12 2022
  read: IOPS=224k, BW=874MiB/s (916MB/s)(6117MiB/7001msec)
    slat (nsec): min=1463, max=42490, avg=1879.75, stdev=476.78
    clat (usec): min=2, max=250, avg=33.63, stdev= 9.41
     lat (usec): min=15, max=262, avg=35.51, stdev= 9.40
    clat percentiles (usec):
     |  1.00th=[   23],  5.00th=[   26], 10.00th=[   28], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   31], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   34], 80.00th=[   37], 90.00th=[   46], 95.00th=[   53],
     | 99.00th=[   64], 99.50th=[   70], 99.90th=[  135], 99.95th=[  143],
     | 99.99th=[  161]
   bw (  KiB/s): min=887864, max=897378, per=100.00%, avg=895042.93, stdev=2300.39, samples=14
   iops        : min=221966, max=224344, avg=223760.50, stdev=575.00, samples=14
  lat (usec)   : 4=0.01%, 20=0.41%, 50=92.44%, 100=6.88%, 250=0.27%
  lat (usec)   : 500=0.01%
  cpu          : usr=22.09%, sys=54.37%, ctx=539751, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1565827,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (916MB/s), 874MiB/s-874MiB/s (916MB/s-916MB/s), io=6117MiB (6414MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_8_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734866,
  "timestamp_ms" : 1666734866018,
  "time" : "Tue Oct 25 23:54:26 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6422921216,
        "io_kbytes" : 6272384,
        "bw_bytes" : 917429112,
        "bw" : 895926,
        "iops" : 223980.717040,
        "runtime" : 7001,
        "total_ios" : 1568089,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1482,
          "max" : 25568,
          "mean" : 1774.253431,
          "stddev" : 161.376328,
          "N" : 1568089
        },
        "clat_ns" : {
          "min" : 13826,
          "max" : 201578,
          "mean" : 33675.984157,
          "stddev" : 9110.646423,
          "N" : 1568096,
          "percentile" : {
            "1.000000" : 23936,
            "5.000000" : 25984,
            "10.000000" : 27264,
            "20.000000" : 29056,
            "30.000000" : 30080,
            "40.000000" : 30848,
            "50.000000" : 31360,
            "60.000000" : 32128,
            "70.000000" : 33024,
            "80.000000" : 35072,
            "90.000000" : 44800,
            "95.000000" : 52480,
            "99.000000" : 62720,
            "99.500000" : 68096,
            "99.900000" : 130560,
            "99.950000" : 142336,
            "99.990000" : 158720
          }
        },
        "lat_ns" : {
          "min" : 15499,
          "max" : 207278,
          "mean" : 35450.237329,
          "stddev" : 9103.650064,
          "N" : 1568096
        },
        "bw_min" : 894984,
        "bw_max" : 897680,
        "bw_agg" : 100.000000,
        "bw_mean" : 896217.000000,
        "bw_dev" : 865.985877,
        "bw_samples" : 14,
        "iops_min" : 223746,
        "iops_max" : 224420,
        "iops_mean" : 224054.214286,
        "iops_stddev" : 216.539276,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 19.214286,
      "sys_cpu" : 54.914286,
      "ctx" : 652730,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.291055,
        "50" : 92.258539,
        "100" : 7.203864,
        "250" : 0.246989,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90091: Tue Oct 25 23:54:26 2022
  read: IOPS=224k, BW=875MiB/s (917MB/s)(6125MiB/7001msec)
    slat (nsec): min=1482, max=25568, avg=1774.25, stdev=161.38
    clat (usec): min=13, max=201, avg=33.68, stdev= 9.11
     lat (usec): min=15, max=207, avg=35.45, stdev= 9.10
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   26], 10.00th=[   28], 20.00th=[   30],
     | 30.00th=[   31], 40.00th=[   31], 50.00th=[   32], 60.00th=[   33],
     | 70.00th=[   34], 80.00th=[   36], 90.00th=[   45], 95.00th=[   53],
     | 99.00th=[   63], 99.50th=[   69], 99.90th=[  131], 99.95th=[  143],
     | 99.99th=[  159]
   bw (  KiB/s): min=894984, max=897680, per=100.00%, avg=896217.00, stdev=865.99, samples=14
   iops        : min=223746, max=224420, avg=224054.21, stdev=216.54, samples=14
  lat (usec)   : 20=0.29%, 50=92.26%, 100=7.20%, 250=0.25%
  cpu          : usr=19.21%, sys=54.91%, ctx=652730, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1568089,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=875MiB/s (917MB/s), 875MiB/s-875MiB/s (917MB/s-917MB/s), io=6125MiB (6423MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_8_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_8_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="8" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_8_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_8_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=8
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666734879,
  "timestamp_ms" : 1666734879675,
  "time" : "Tue Oct 25 23:54:39 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "8",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 6417117184,
        "io_kbytes" : 6266716,
        "bw_bytes" : 916600083,
        "bw" : 895117,
        "iops" : 223778.317383,
        "runtime" : 7001,
        "total_ios" : 1566672,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1452,
          "max" : 42480,
          "mean" : 2043.000890,
          "stddev" : 749.595984,
          "N" : 1566672
        },
        "clat_ns" : {
          "min" : 440,
          "max" : 228789,
          "mean" : 33399.843687,
          "stddev" : 8968.879387,
          "N" : 1566679,
          "percentile" : {
            "1.000000" : 23168,
            "5.000000" : 27008,
            "10.000000" : 27520,
            "20.000000" : 28544,
            "30.000000" : 29824,
            "40.000000" : 31104,
            "50.000000" : 31360,
            "60.000000" : 31616,
            "70.000000" : 32384,
            "80.000000" : 34048,
            "90.000000" : 44800,
            "95.000000" : 51968,
            "99.000000" : 62208,
            "99.500000" : 67072,
            "99.900000" : 132096,
            "99.950000" : 140288,
            "99.990000" : 156672
          }
        },
        "lat_ns" : {
          "min" : 18515,
          "max" : 230592,
          "mean" : 35442.843629,
          "stddev" : 8913.795051,
          "N" : 1566679
        },
        "bw_min" : 893096,
        "bw_max" : 897528,
        "bw_agg" : 100.000000,
        "bw_mean" : 895350.214286,
        "bw_dev" : 1168.143970,
        "bw_samples" : 14,
        "iops_min" : 223274,
        "iops_max" : 224382,
        "iops_mean" : 223837.500000,
        "iops_stddev" : 292.004413,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 23.857143,
      "sys_cpu" : 59.200000,
      "ctx" : 370799,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 0.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 100.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 99.999968,
        "8" : 0.100000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 0.034468,
        "50" : 92.821854,
        "100" : 6.875274,
        "250" : 0.268595,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 8,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=90132: Tue Oct 25 23:54:39 2022
  read: IOPS=224k, BW=874MiB/s (917MB/s)(6120MiB/7001msec)
    slat (nsec): min=1452, max=42480, avg=2043.00, stdev=749.60
    clat (nsec): min=440, max=228789, avg=33399.84, stdev=8968.88
     lat (usec): min=18, max=230, avg=35.44, stdev= 8.91
    clat percentiles (usec):
     |  1.00th=[   24],  5.00th=[   28], 10.00th=[   28], 20.00th=[   29],
     | 30.00th=[   30], 40.00th=[   32], 50.00th=[   32], 60.00th=[   32],
     | 70.00th=[   33], 80.00th=[   35], 90.00th=[   45], 95.00th=[   52],
     | 99.00th=[   63], 99.50th=[   68], 99.90th=[  133], 99.95th=[  141],
     | 99.99th=[  157]
   bw (  KiB/s): min=893096, max=897528, per=100.00%, avg=895350.21, stdev=1168.14, samples=14
   iops        : min=223274, max=224382, avg=223837.50, stdev=292.00, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=0.03%, 50=92.82%, 100=6.88%
  lat (usec)   : 250=0.27%
  cpu          : usr=23.86%, sys=59.20%, ctx=370799, majf=0, minf=0
  IO depths    : 1=0.0%, 2=0.0%, 4=0.0%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1566672,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
   READ: bw=874MiB/s (917MB/s), 874MiB/s-874MiB/s (917MB/s-917MB/s), io=6120MiB (6417MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_1.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733187,
  "timestamp_ms" : 1666733187146,
  "time" : "Tue Oct 25 23:26:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2400022528,
        "io_kbytes" : 2343772,
        "bw_bytes" : 342811388,
        "bw" : 334776,
        "iops" : 83694.186545,
        "runtime" : 7001,
        "total_ios" : 585943,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 160,
          "max" : 9938,
          "mean" : 176.388493,
          "stddev" : 26.408079,
          "N" : 585943
        },
        "clat_ns" : {
          "min" : 10430,
          "max" : 113473,
          "mean" : 11527.381930,
          "stddev" : 2234.725223,
          "N" : 585943,
          "percentile" : {
            "1.000000" : 10944,
            "5.000000" : 11072,
            "10.000000" : 11072,
            "20.000000" : 11072,
            "30.000000" : 11072,
            "40.000000" : 11072,
            "50.000000" : 11200,
            "60.000000" : 11200,
            "70.000000" : 11328,
            "80.000000" : 11328,
            "90.000000" : 11456,
            "95.000000" : 12096,
            "99.000000" : 22400,
            "99.500000" : 28288,
            "99.900000" : 28800,
            "99.950000" : 44288,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 10610,
          "max" : 113653,
          "mean" : 11703.770423,
          "stddev" : 2235.118972,
          "N" : 585943
        },
        "bw_min" : 334472,
        "bw_max" : 336208,
        "bw_agg" : 100.000000,
        "bw_mean" : 334961.142857,
        "bw_dev" : 434.449228,
        "bw_samples" : 14,
        "iops_min" : 83618,
        "iops_max" : 84052,
        "iops_mean" : 83740.214286,
        "iops_stddev" : 108.614476,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 80.728571,
      "sys_cpu" : 19.257143,
      "ctx" : 15,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 98.614541,
        "50" : 1.340915,
        "100" : 0.044373,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84947: Tue Oct 25 23:26:27 2022
  read: IOPS=83.7k, BW=327MiB/s (343MB/s)(2289MiB/7001msec)
    slat (nsec): min=160, max=9938, avg=176.39, stdev=26.41
    clat (usec): min=10, max=113, avg=11.53, stdev= 2.23
     lat (usec): min=10, max=113, avg=11.70, stdev= 2.24
    clat percentiles (nsec):
     |  1.00th=[10944],  5.00th=[11072], 10.00th=[11072], 20.00th=[11072],
     | 30.00th=[11072], 40.00th=[11072], 50.00th=[11200], 60.00th=[11200],
     | 70.00th=[11328], 80.00th=[11328], 90.00th=[11456], 95.00th=[12096],
     | 99.00th=[22400], 99.50th=[28288], 99.90th=[28800], 99.95th=[44288],
     | 99.99th=[68096]
   bw (  KiB/s): min=334472, max=336208, per=100.00%, avg=334961.14, stdev=434.45, samples=14
   iops        : min=83618, max=84052, avg=83740.21, stdev=108.61, samples=14
  lat (usec)   : 20=98.61%, 50=1.34%, 100=0.04%, 250=0.01%
  cpu          : usr=80.73%, sys=19.26%, ctx=15, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=585943,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=327MiB/s (343MB/s), 327MiB/s-327MiB/s (343MB/s-343MB/s), io=2289MiB (2400MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_uring" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_uring_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring-bdev_uring_2.txt
io_uring-bdev_uring: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733202,
  "timestamp_ms" : 1666733202256,
  "time" : "Tue Oct 25 23:26:42 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_uring",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_uring",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_uring_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2856644608,
        "io_kbytes" : 2789692,
        "bw_bytes" : 408033796,
        "bw" : 398470,
        "iops" : 99617.626053,
        "runtime" : 7001,
        "total_ios" : 697423,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 161,
          "max" : 12935,
          "mean" : 181.008962,
          "stddev" : 43.708701,
          "N" : 697423
        },
        "clat_ns" : {
          "min" : 9037,
          "max" : 95539,
          "mean" : 9612.561520,
          "stddev" : 1943.039396,
          "N" : 697423,
          "percentile" : {
            "1.000000" : 9152,
            "5.000000" : 9152,
            "10.000000" : 9152,
            "20.000000" : 9152,
            "30.000000" : 9280,
            "40.000000" : 9280,
            "50.000000" : 9280,
            "60.000000" : 9408,
            "70.000000" : 9408,
            "80.000000" : 9408,
            "90.000000" : 9408,
            "95.000000" : 10048,
            "99.000000" : 18816,
            "99.500000" : 20864,
            "99.900000" : 25472,
            "99.950000" : 42752,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 9218,
          "max" : 95719,
          "mean" : 9793.570482,
          "stddev" : 1943.679350,
          "N" : 697423
        },
        "bw_min" : 397920,
        "bw_max" : 399024,
        "bw_agg" : 100.000000,
        "bw_mean" : 398570.642857,
        "bw_dev" : 358.930807,
        "bw_samples" : 14,
        "iops_min" : 99480,
        "iops_max" : 99756,
        "iops_mean" : 99642.642857,
        "iops_stddev" : 89.712024,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 70.271429,
      "sys_cpu" : 29.700000,
      "ctx" : 8,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 94.961594,
        "20" : 4.229141,
        "50" : 0.766393,
        "100" : 0.042872,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_uring: (groupid=0, jobs=1): err= 0: pid=84996: Tue Oct 25 23:26:42 2022
  read: IOPS=99.6k, BW=389MiB/s (408MB/s)(2724MiB/7001msec)
    slat (nsec): min=161, max=12935, avg=181.01, stdev=43.71
    clat (nsec): min=9037, max=95539, avg=9612.56, stdev=1943.04
     lat (nsec): min=9218, max=95719, avg=9793.57, stdev=1943.68
    clat percentiles (nsec):
     |  1.00th=[ 9152],  5.00th=[ 9152], 10.00th=[ 9152], 20.00th=[ 9152],
     | 30.00th=[ 9280], 40.00th=[ 9280], 50.00th=[ 9280], 60.00th=[ 9408],
     | 70.00th=[ 9408], 80.00th=[ 9408], 90.00th=[ 9408], 95.00th=[10048],
     | 99.00th=[18816], 99.50th=[20864], 99.90th=[25472], 99.95th=[42752],
     | 99.99th=[67072]
   bw (  KiB/s): min=397920, max=399024, per=100.00%, avg=398570.64, stdev=358.93, samples=14
   iops        : min=99480, max=99756, avg=99642.64, stdev=89.71, samples=14
  lat (usec)   : 10=94.96%, 20=4.23%, 50=0.77%, 100=0.04%
  cpu          : usr=70.27%, sys=29.70%, ctx=8, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=697423,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=389MiB/s (408MB/s), 389MiB/s-389MiB/s (408MB/s-408MB/s), io=2724MiB (2857MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_0.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733217,
  "timestamp_ms" : 1666733217255,
  "time" : "Tue Oct 25 23:26:57 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2650079232,
        "io_kbytes" : 2587968,
        "bw_bytes" : 378528671,
        "bw" : 369656,
        "iops" : 92414.226539,
        "runtime" : 7001,
        "total_ios" : 646992,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1673,
          "max" : 15529,
          "mean" : 1800.539421,
          "stddev" : 116.222659,
          "N" : 646992
        },
        "clat_ns" : {
          "min" : 251,
          "max" : 115536,
          "mean" : 8778.187362,
          "stddev" : 1981.005506,
          "N" : 646992,
          "percentile" : {
            "1.000000" : 8256,
            "5.000000" : 8384,
            "10.000000" : 8384,
            "20.000000" : 8384,
            "30.000000" : 8384,
            "40.000000" : 8384,
            "50.000000" : 8512,
            "60.000000" : 8512,
            "70.000000" : 8512,
            "80.000000" : 8512,
            "90.000000" : 8640,
            "95.000000" : 9408,
            "99.000000" : 18816,
            "99.500000" : 20352,
            "99.900000" : 24960,
            "99.950000" : 41728,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 9598,
          "max" : 117290,
          "mean" : 10578.726783,
          "stddev" : 1983.792726,
          "N" : 646992
        },
        "bw_min" : 367720,
        "bw_max" : 370808,
        "bw_agg" : 100.000000,
        "bw_mean" : 369810.769231,
        "bw_dev" : 877.410504,
        "bw_samples" : 13,
        "iops_min" : 91930,
        "iops_max" : 92702,
        "iops_mean" : 92452.615385,
        "iops_stddev" : 219.323406,
        "iops_samples" : 13
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 5.642857,
      "sys_cpu" : 94.328571,
      "ctx" : 14,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.804121,
        "20" : 3.551049,
        "50" : 0.603562,
        "100" : 0.036940,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85046: Tue Oct 25 23:26:57 2022
  read: IOPS=92.4k, BW=361MiB/s (379MB/s)(2527MiB/7001msec)
    slat (nsec): min=1673, max=15529, avg=1800.54, stdev=116.22
    clat (nsec): min=251, max=115536, avg=8778.19, stdev=1981.01
     lat (usec): min=9, max=117, avg=10.58, stdev= 1.98
    clat percentiles (nsec):
     |  1.00th=[ 8256],  5.00th=[ 8384], 10.00th=[ 8384], 20.00th=[ 8384],
     | 30.00th=[ 8384], 40.00th=[ 8384], 50.00th=[ 8512], 60.00th=[ 8512],
     | 70.00th=[ 8512], 80.00th=[ 8512], 90.00th=[ 8640], 95.00th=[ 9408],
     | 99.00th=[18816], 99.50th=[20352], 99.90th=[24960], 99.95th=[41728],
     | 99.99th=[66048]
   bw (  KiB/s): min=367720, max=370808, per=100.00%, avg=369810.77, stdev=877.41, samples=13
   iops        : min=91930, max=92702, avg=92452.62, stdev=219.32, samples=13
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=95.80%, 20=3.55%, 50=0.60%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=5.64%, sys=94.33%, ctx=14, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=646992,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=361MiB/s (379MB/s), 361MiB/s-361MiB/s (379MB/s-379MB/s), io=2527MiB (2650MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_1.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733232,
  "timestamp_ms" : 1666733232288,
  "time" : "Tue Oct 25 23:27:12 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2934267904,
        "io_kbytes" : 2865496,
        "bw_bytes" : 419121254,
        "bw" : 409298,
        "iops" : 102324.525068,
        "runtime" : 7001,
        "total_ios" : 716374,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1543,
          "max" : 18425,
          "mean" : 1610.136684,
          "stddev" : 115.405056,
          "N" : 716374
        },
        "clat_ns" : {
          "min" : 400,
          "max" : 1295729,
          "mean" : 7923.244064,
          "stddev" : 2451.556179,
          "N" : 716374,
          "percentile" : {
            "1.000000" : 7392,
            "5.000000" : 7456,
            "10.000000" : 7456,
            "20.000000" : 7520,
            "30.000000" : 7520,
            "40.000000" : 7584,
            "50.000000" : 7648,
            "60.000000" : 7648,
            "70.000000" : 7712,
            "80.000000" : 7712,
            "90.000000" : 7776,
            "95.000000" : 8160,
            "99.000000" : 17280,
            "99.500000" : 20096,
            "99.900000" : 22656,
            "99.950000" : 32384,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 8926,
          "max" : 1297322,
          "mean" : 9533.380748,
          "stddev" : 2454.613923,
          "N" : 716374
        },
        "bw_min" : 408576,
        "bw_max" : 410869,
        "bw_agg" : 100.000000,
        "bw_mean" : 409391.785714,
        "bw_dev" : 583.681180,
        "bw_samples" : 14,
        "iops_min" : 102144,
        "iops_max" : 102717,
        "iops_mean" : 102347.928571,
        "iops_stddev" : 145.871632,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 6.957143,
      "sys_cpu" : 93.000000,
      "ctx" : 48,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.128419,
        "20" : 3.324520,
        "50" : 0.511325,
        "100" : 0.034619,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85097: Tue Oct 25 23:27:12 2022
  read: IOPS=102k, BW=400MiB/s (419MB/s)(2798MiB/7001msec)
    slat (nsec): min=1543, max=18425, avg=1610.14, stdev=115.41
    clat (nsec): min=400, max=1295.7k, avg=7923.24, stdev=2451.56
     lat (usec): min=8, max=1297, avg= 9.53, stdev= 2.45
    clat percentiles (nsec):
     |  1.00th=[ 7392],  5.00th=[ 7456], 10.00th=[ 7456], 20.00th=[ 7520],
     | 30.00th=[ 7520], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 7648],
     | 70.00th=[ 7712], 80.00th=[ 7712], 90.00th=[ 7776], 95.00th=[ 8160],
     | 99.00th=[17280], 99.50th=[20096], 99.90th=[22656], 99.95th=[32384],
     | 99.99th=[65280]
   bw (  KiB/s): min=408576, max=410869, per=100.00%, avg=409391.79, stdev=583.68, samples=14
   iops        : min=102144, max=102717, avg=102347.93, stdev=145.87, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.13%, 20=3.32%, 50=0.51%, 100=0.03%
  lat (usec)   : 250=0.01%
  lat (msec)   : 2=0.01%
  cpu          : usr=6.96%, sys=93.00%, ctx=48, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=716374,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=400MiB/s (419MB/s), 400MiB/s-400MiB/s (419MB/s-419MB/s), io=2798MiB (2934MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_io_uring.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_io_uring-bdev_xnvme_2.txt
io_uring-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733247,
  "timestamp_ms" : 1666733247334,
  "time" : "Tue Oct 25 23:27:27 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_io_uring.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2939764736,
        "io_kbytes" : 2870864,
        "bw_bytes" : 419906404,
        "bw" : 410064,
        "iops" : 102516.211970,
        "runtime" : 7001,
        "total_ios" : 717716,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1522,
          "max" : 14347,
          "mean" : 1578.864615,
          "stddev" : 120.389006,
          "N" : 717716
        },
        "clat_ns" : {
          "min" : 241,
          "max" : 96561,
          "mean" : 7938.478975,
          "stddev" : 1946.448668,
          "N" : 717716,
          "percentile" : {
            "1.000000" : 7456,
            "5.000000" : 7456,
            "10.000000" : 7456,
            "20.000000" : 7520,
            "30.000000" : 7520,
            "40.000000" : 7584,
            "50.000000" : 7648,
            "60.000000" : 7648,
            "70.000000" : 7712,
            "80.000000" : 7712,
            "90.000000" : 7776,
            "95.000000" : 8160,
            "99.000000" : 17280,
            "99.500000" : 20352,
            "99.900000" : 22912,
            "99.950000" : 32640,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 8926,
          "max" : 98124,
          "mean" : 9517.343590,
          "stddev" : 1950.790720,
          "N" : 717716
        },
        "bw_min" : 409424,
        "bw_max" : 411160,
        "bw_agg" : 100.000000,
        "bw_mean" : 410161.928571,
        "bw_dev" : 477.509318,
        "bw_samples" : 14,
        "iops_min" : 102354,
        "iops_max" : 102790,
        "iops_mean" : 102540.428571,
        "iops_stddev" : 119.741204,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 8.157143,
      "sys_cpu" : 91.814286,
      "ctx" : 5,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.122422,
        "20" : 3.278456,
        "50" : 0.559135,
        "100" : 0.038316,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85148: Tue Oct 25 23:27:27 2022
  read: IOPS=103k, BW=400MiB/s (420MB/s)(2804MiB/7001msec)
    slat (nsec): min=1522, max=14347, avg=1578.86, stdev=120.39
    clat (nsec): min=241, max=96561, avg=7938.48, stdev=1946.45
     lat (nsec): min=8926, max=98124, avg=9517.34, stdev=1950.79
    clat percentiles (nsec):
     |  1.00th=[ 7456],  5.00th=[ 7456], 10.00th=[ 7456], 20.00th=[ 7520],
     | 30.00th=[ 7520], 40.00th=[ 7584], 50.00th=[ 7648], 60.00th=[ 7648],
     | 70.00th=[ 7712], 80.00th=[ 7712], 90.00th=[ 7776], 95.00th=[ 8160],
     | 99.00th=[17280], 99.50th=[20352], 99.90th=[22912], 99.95th=[32640],
     | 99.99th=[65280]
   bw (  KiB/s): min=409424, max=411160, per=100.00%, avg=410161.93, stdev=477.51, samples=14
   iops        : min=102354, max=102790, avg=102540.43, stdev=119.74, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%, 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.12%, 20=3.28%, 50=0.56%, 100=0.04%
  cpu          : usr=8.16%, sys=91.81%, ctx=5, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=717716,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=400MiB/s (420MB/s), 400MiB/s-400MiB/s (420MB/s-420MB/s), io=2804MiB (2940MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_0.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733261,
  "timestamp_ms" : 1666733261012,
  "time" : "Tue Oct 25 23:27:41 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1795207168,
        "io_kbytes" : 1753132,
        "bw_bytes" : 256421535,
        "bw" : 250411,
        "iops" : 62602.913869,
        "runtime" : 7001,
        "total_ios" : 438283,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2655,
          "max" : 27832,
          "mean" : 2842.061771,
          "stddev" : 230.543540,
          "N" : 438283
        },
        "clat_ns" : {
          "min" : 281,
          "max" : 122980,
          "mean" : 12554.664060,
          "stddev" : 2139.295770,
          "N" : 438283,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 12096,
            "10.000000" : 12096,
            "20.000000" : 12096,
            "30.000000" : 12224,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12480,
            "95.000000" : 12608,
            "99.000000" : 22144,
            "99.500000" : 28288,
            "99.900000" : 40704,
            "99.950000" : 54016,
            "99.990000" : 68096
          }
        },
        "lat_ns" : {
          "min" : 12182,
          "max" : 125816,
          "mean" : 15396.725830,
          "stddev" : 2157.279613,
          "N" : 438283
        },
        "bw_min" : 249896,
        "bw_max" : 251094,
        "bw_agg" : 100.000000,
        "bw_mean" : 250470.142857,
        "bw_dev" : 335.320250,
        "bw_samples" : 14,
        "iops_min" : 62472,
        "iops_max" : 62773,
        "iops_mean" : 62617.500000,
        "iops_stddev" : 83.901728,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.585714,
      "sys_cpu" : 38.442857,
      "ctx" : 438270,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.596113,
        "50" : 1.331788,
        "100" : 0.064798,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85187: Tue Oct 25 23:27:41 2022
  read: IOPS=62.6k, BW=245MiB/s (256MB/s)(1712MiB/7001msec)
    slat (nsec): min=2655, max=27832, avg=2842.06, stdev=230.54
    clat (nsec): min=281, max=122980, avg=12554.66, stdev=2139.30
     lat (usec): min=12, max=125, avg=15.40, stdev= 2.16
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[12096], 10.00th=[12096], 20.00th=[12096],
     | 30.00th=[12224], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
     | 99.00th=[22144], 99.50th=[28288], 99.90th=[40704], 99.95th=[54016],
     | 99.99th=[68096]
   bw (  KiB/s): min=249896, max=251094, per=100.00%, avg=250470.14, stdev=335.32, samples=14
   iops        : min=62472, max=62773, avg=62617.50, stdev=83.90, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.60%, 50=1.33%, 100=0.06%
  lat (usec)   : 250=0.01%
  cpu          : usr=10.59%, sys=38.44%, ctx=438270, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=438283,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (256MB/s), 245MiB/s-245MiB/s (256MB/s-256MB/s), io=1712MiB (1795MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_1.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_1.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733274,
  "timestamp_ms" : 1666733274648,
  "time" : "Tue Oct 25 23:27:54 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1804898304,
        "io_kbytes" : 1762596,
        "bw_bytes" : 257805785,
        "bw" : 251763,
        "iops" : 62940.865591,
        "runtime" : 7001,
        "total_ios" : 440649,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2645,
          "max" : 27261,
          "mean" : 2816.685162,
          "stddev" : 258.194088,
          "N" : 440649
        },
        "clat_ns" : {
          "min" : 380,
          "max" : 467586,
          "mean" : 12522.016850,
          "stddev" : 2175.039916,
          "N" : 440649,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 12096,
            "10.000000" : 12096,
            "20.000000" : 12096,
            "30.000000" : 12224,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12352,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12480,
            "95.000000" : 12608,
            "99.000000" : 22144,
            "99.500000" : 28032,
            "99.900000" : 33536,
            "99.950000" : 50432,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 12243,
          "max" : 470382,
          "mean" : 15338.702012,
          "stddev" : 2192.213103,
          "N" : 440649
        },
        "bw_min" : 250829,
        "bw_max" : 252600,
        "bw_agg" : 100.000000,
        "bw_mean" : 251826.071429,
        "bw_dev" : 417.064921,
        "bw_samples" : 14,
        "iops_min" : 62707,
        "iops_max" : 63150,
        "iops_mean" : 62956.500000,
        "iops_stddev" : 104.312216,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.042857,
      "sys_cpu" : 38.328571,
      "ctx" : 440630,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.638372,
        "50" : 1.295589,
        "100" : 0.056054,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85227: Tue Oct 25 23:27:54 2022
  read: IOPS=62.9k, BW=246MiB/s (258MB/s)(1721MiB/7001msec)
    slat (nsec): min=2645, max=27261, avg=2816.69, stdev=258.19
    clat (nsec): min=380, max=467586, avg=12522.02, stdev=2175.04
     lat (usec): min=12, max=470, avg=15.34, stdev= 2.19
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[12096], 10.00th=[12096], 20.00th=[12096],
     | 30.00th=[12224], 40.00th=[12224], 50.00th=[12224], 60.00th=[12352],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
     | 99.00th=[22144], 99.50th=[28032], 99.90th=[33536], 99.95th=[50432],
     | 99.99th=[66048]
   bw (  KiB/s): min=250829, max=252600, per=100.00%, avg=251826.07, stdev=417.06, samples=14
   iops        : min=62707, max=63150, avg=62956.50, stdev=104.31, samples=14
  lat (nsec)   : 500=0.01%
  lat (usec)   : 4=0.01%, 10=0.01%, 20=98.64%, 50=1.30%, 100=0.06%
  lat (usec)   : 250=0.01%, 500=0.01%
  cpu          : usr=11.04%, sys=38.33%, ctx=440630, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=440649,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=246MiB/s (258MB/s), 246MiB/s-246MiB/s (258MB/s-258MB/s), io=1721MiB (1805MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_io_uring-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_io_uring-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="io_uring-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="io_uring" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_io_uring-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_io_uring-xnvme_2.txt
io_uring-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733288,
  "timestamp_ms" : 1666733288284,
  "time" : "Tue Oct 25 23:28:08 2022",
  "jobs" : [
    {
      "jobname" : "io_uring-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "io_uring-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "io_uring",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1813983232,
        "io_kbytes" : 1771468,
        "bw_bytes" : 259103446,
        "bw" : 253030,
        "iops" : 63257.534638,
        "runtime" : 7001,
        "total_ios" : 442866,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2584,
          "max" : 28093,
          "mean" : 2753.253893,
          "stddev" : 286.498944,
          "N" : 442867
        },
        "clat_ns" : {
          "min" : 421,
          "max" : 181340,
          "mean" : 12481.790944,
          "stddev" : 2028.063473,
          "N" : 442866,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 11968,
            "10.000000" : 11968,
            "20.000000" : 12096,
            "30.000000" : 12096,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12224,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12352,
            "95.000000" : 12608,
            "99.000000" : 22144,
            "99.500000" : 27776,
            "99.900000" : 32384,
            "99.950000" : 50432,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 12102,
          "max" : 184025,
          "mean" : 15235.045014,
          "stddev" : 2052.149286,
          "N" : 442866
        },
        "bw_min" : 252120,
        "bw_max" : 254468,
        "bw_agg" : 100.000000,
        "bw_mean" : 253092.857143,
        "bw_dev" : 627.819040,
        "bw_samples" : 14,
        "iops_min" : 63030,
        "iops_max" : 63617,
        "iops_mean" : 63273.214286,
        "iops_stddev" : 156.954760,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.757143,
      "sys_cpu" : 38.157143,
      "ctx" : 442837,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.010000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.632995,
        "50" : 1.304909,
        "100" : 0.052612,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

io_uring-xnvme: (groupid=0, jobs=1): err= 0: pid=85266: Tue Oct 25 23:28:08 2022
  read: IOPS=63.3k, BW=247MiB/s (259MB/s)(1730MiB/7001msec)
    slat (nsec): min=2584, max=28093, avg=2753.25, stdev=286.50
    clat (nsec): min=421, max=181340, avg=12481.79, stdev=2028.06
     lat (usec): min=12, max=184, avg=15.24, stdev= 2.05
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[11968], 10.00th=[11968], 20.00th=[12096],
     | 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12224],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12352], 95.00th=[12608],
     | 99.00th=[22144], 99.50th=[27776], 99.90th=[32384], 99.95th=[50432],
     | 99.99th=[66048]
   bw (  KiB/s): min=252120, max=254468, per=100.00%, avg=253092.86, stdev=627.82, samples=14
   iops        : min=63030, max=63617, avg=63273.21, stdev=156.95, samples=14
  lat (nsec)   : 500=0.01%, 750=0.01%
  lat (usec)   : 10=0.01%, 20=98.63%, 50=1.30%, 100=0.05%, 250=0.01%
  cpu          : usr=10.76%, sys=38.16%, ctx=442837, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=442866,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=247MiB/s (259MB/s), 247MiB/s-247MiB/s (259MB/s-259MB/s), io=1730MiB (1814MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-reference_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_0.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_0.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733301,
  "timestamp_ms" : 1666733301938,
  "time" : "Tue Oct 25 23:28:21 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1673572352,
        "io_kbytes" : 1634348,
        "bw_bytes" : 239047614,
        "bw" : 233444,
        "iops" : 58361.234109,
        "runtime" : 7001,
        "total_ios" : 408587,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2836,
          "max" : 17292,
          "mean" : 3074.658611,
          "stddev" : 158.340253,
          "N" : 408587
        },
        "clat_ns" : {
          "min" : 802,
          "max" : 118943,
          "mean" : 13512.992719,
          "stddev" : 2095.582789,
          "N" : 408587,
          "percentile" : {
            "1.000000" : 12864,
            "5.000000" : 12992,
            "10.000000" : 13120,
            "20.000000" : 13120,
            "30.000000" : 13248,
            "40.000000" : 13248,
            "50.000000" : 13248,
            "60.000000" : 13248,
            "70.000000" : 13248,
            "80.000000" : 13376,
            "90.000000" : 13376,
            "95.000000" : 13504,
            "99.000000" : 23680,
            "99.500000" : 28800,
            "99.900000" : 33536,
            "99.950000" : 50432,
            "99.990000" : 71168
          }
        },
        "lat_ns" : {
          "min" : 14397,
          "max" : 122139,
          "mean" : 16587.651330,
          "stddev" : 2102.634447,
          "N" : 408587
        },
        "bw_min" : 233168,
        "bw_max" : 233976,
        "bw_agg" : 100.000000,
        "bw_mean" : 233596.714286,
        "bw_dev" : 275.558574,
        "bw_samples" : 14,
        "iops_min" : 58292,
        "iops_max" : 58494,
        "iops_mean" : 58399.000000,
        "iops_stddev" : 68.835367,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 9.100000,
      "sys_cpu" : 32.528571,
      "ctx" : 408588,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.564810,
        "50" : 1.376696,
        "100" : 0.051641,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 586804,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5712,
      "write_ticks" : 0,
      "in_queue" : 5712,
      "util" : 99.108381
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=85307: Tue Oct 25 23:28:21 2022
  read: IOPS=58.4k, BW=228MiB/s (239MB/s)(1596MiB/7001msec)
    slat (nsec): min=2836, max=17292, avg=3074.66, stdev=158.34
    clat (nsec): min=802, max=118943, avg=13512.99, stdev=2095.58
     lat (usec): min=14, max=122, avg=16.59, stdev= 2.10
    clat percentiles (nsec):
     |  1.00th=[12864],  5.00th=[12992], 10.00th=[13120], 20.00th=[13120],
     | 30.00th=[13248], 40.00th=[13248], 50.00th=[13248], 60.00th=[13248],
     | 70.00th=[13248], 80.00th=[13376], 90.00th=[13376], 95.00th=[13504],
     | 99.00th=[23680], 99.50th=[28800], 99.90th=[33536], 99.95th=[50432],
     | 99.99th=[71168]
   bw (  KiB/s): min=233168, max=233976, per=100.00%, avg=233596.71, stdev=275.56, samples=14
   iops        : min=58292, max=58494, avg=58399.00, stdev=68.84, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 10=0.01%, 20=98.56%, 50=1.38%, 100=0.05%, 250=0.01%
  cpu          : usr=9.10%, sys=32.53%, ctx=408588, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=408587,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=228MiB/s (239MB/s), 228MiB/s-228MiB/s (239MB/s-239MB/s), io=1596MiB (1674MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=586804/0, merge=0/0, ticks=5712/0, in_queue=5712, util=99.11%

$ rm /tmp/fio-output_4K_1_libaio-reference_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_1.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_1.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733315,
  "timestamp_ms" : 1666733315587,
  "time" : "Tue Oct 25 23:28:35 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1801781248,
        "io_kbytes" : 1759552,
        "bw_bytes" : 257360555,
        "bw" : 251328,
        "iops" : 62832.166833,
        "runtime" : 7001,
        "total_ios" : 439888,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2765,
          "max" : 26921,
          "mean" : 2932.077167,
          "stddev" : 192.262685,
          "N" : 439888
        },
        "clat_ns" : {
          "min" : 881,
          "max" : 77566,
          "mean" : 12358.918272,
          "stddev" : 1970.343765,
          "N" : 439888,
          "percentile" : {
            "1.000000" : 11840,
            "5.000000" : 11840,
            "10.000000" : 11968,
            "20.000000" : 11968,
            "30.000000" : 11968,
            "40.000000" : 12096,
            "50.000000" : 12096,
            "60.000000" : 12096,
            "70.000000" : 12224,
            "80.000000" : 12224,
            "90.000000" : 12352,
            "95.000000" : 12480,
            "99.000000" : 21888,
            "99.500000" : 27264,
            "99.900000" : 31616,
            "99.950000" : 49920,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 12002,
          "max" : 80582,
          "mean" : 15290.995440,
          "stddev" : 1982.252033,
          "N" : 439888
        },
        "bw_min" : 250264,
        "bw_max" : 252440,
        "bw_agg" : 100.000000,
        "bw_mean" : 251431.928571,
        "bw_dev" : 631.720540,
        "bw_samples" : 14,
        "iops_min" : 62566,
        "iops_max" : 63110,
        "iops_mean" : 62857.928571,
        "iops_stddev" : 157.898928,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 11.057143,
      "sys_cpu" : 37.400000,
      "ctx" : 439880,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010000,
        "20" : 98.638062,
        "50" : 1.307151,
        "100" : 0.047057,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 631256,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 5943,
      "write_ticks" : 0,
      "in_queue" : 5943,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=85345: Tue Oct 25 23:28:35 2022
  read: IOPS=62.8k, BW=245MiB/s (257MB/s)(1718MiB/7001msec)
    slat (nsec): min=2765, max=26921, avg=2932.08, stdev=192.26
    clat (nsec): min=881, max=77566, avg=12358.92, stdev=1970.34
     lat (nsec): min=12002, max=80582, avg=15291.00, stdev=1982.25
    clat percentiles (nsec):
     |  1.00th=[11840],  5.00th=[11840], 10.00th=[11968], 20.00th=[11968],
     | 30.00th=[11968], 40.00th=[12096], 50.00th=[12096], 60.00th=[12096],
     | 70.00th=[12224], 80.00th=[12224], 90.00th=[12352], 95.00th=[12480],
     | 99.00th=[21888], 99.50th=[27264], 99.90th=[31616], 99.95th=[49920],
     | 99.99th=[67072]
   bw (  KiB/s): min=250264, max=252440, per=100.00%, avg=251431.93, stdev=631.72, samples=14
   iops        : min=62566, max=63110, avg=62857.93, stdev=157.90, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 10=0.01%, 20=98.64%, 50=1.31%, 100=0.05%
  cpu          : usr=11.06%, sys=37.40%, ctx=439880, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=439888,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=245MiB/s (257MB/s), 245MiB/s-245MiB/s (257MB/s-257MB/s), io=1718MiB (1802MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=631256/0, merge=0/0, ticks=5943/0, in_queue=5943, util=99.09%

$ rm /tmp/fio-output_4K_1_libaio-reference_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-reference_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-reference" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="libaio" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-reference_2.txt"
$ cat /tmp/fio-output_4K_1_libaio-reference_2.txt
libaio-reference: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733329,
  "timestamp_ms" : 1666733329246,
  "time" : "Tue Oct 25 23:28:49 2022",
  "jobs" : [
    {
      "jobname" : "libaio-reference",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-reference",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "libaio",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1831936000,
        "io_kbytes" : 1789000,
        "bw_bytes" : 261667761,
        "bw" : 255534,
        "iops" : 63883.588059,
        "runtime" : 7001,
        "total_ios" : 447249,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2605,
          "max" : 37320,
          "mean" : 2851.803557,
          "stddev" : 243.670282,
          "N" : 447250
        },
        "clat_ns" : {
          "min" : 872,
          "max" : 136185,
          "mean" : 12256.182628,
          "stddev" : 1988.047396,
          "N" : 447249,
          "percentile" : {
            "1.000000" : 11712,
            "5.000000" : 11712,
            "10.000000" : 11840,
            "20.000000" : 11840,
            "30.000000" : 11840,
            "40.000000" : 11968,
            "50.000000" : 11968,
            "60.000000" : 12096,
            "70.000000" : 12096,
            "80.000000" : 12096,
            "90.000000" : 12224,
            "95.000000" : 12352,
            "99.000000" : 22144,
            "99.500000" : 26752,
            "99.900000" : 31616,
            "99.950000" : 49408,
            "99.990000" : 63744
          }
        },
        "lat_ns" : {
          "min" : 12163,
          "max" : 138971,
          "mean" : 15107.986178,
          "stddev" : 2008.944842,
          "N" : 447249
        },
        "bw_min" : 254784,
        "bw_max" : 257032,
        "bw_agg" : 100.000000,
        "bw_mean" : 255632.357143,
        "bw_dev" : 623.778265,
        "bw_samples" : 14,
        "iops_min" : 63696,
        "iops_max" : 64258,
        "iops_mean" : 63908.000000,
        "iops_stddev" : 155.974850,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 10.242857,
      "sys_cpu" : 37.271429,
      "ctx" : 447233,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.553155,
        "50" : 1.395867,
        "100" : 0.044047,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ],
  "disk_util" : [
    {
      "name" : "nvme0n1",
      "read_ios" : 640196,
      "write_ios" : 0,
      "read_merges" : 0,
      "write_merges" : 0,
      "read_ticks" : 6020,
      "write_ticks" : 0,
      "in_queue" : 6020,
      "util" : 99.088748
    }
  ]
}

libaio-reference: (groupid=0, jobs=1): err= 0: pid=85383: Tue Oct 25 23:28:49 2022
  read: IOPS=63.9k, BW=250MiB/s (262MB/s)(1747MiB/7001msec)
    slat (nsec): min=2605, max=37320, avg=2851.80, stdev=243.67
    clat (nsec): min=872, max=136185, avg=12256.18, stdev=1988.05
     lat (usec): min=12, max=138, avg=15.11, stdev= 2.01
    clat percentiles (nsec):
     |  1.00th=[11712],  5.00th=[11712], 10.00th=[11840], 20.00th=[11840],
     | 30.00th=[11840], 40.00th=[11968], 50.00th=[11968], 60.00th=[12096],
     | 70.00th=[12096], 80.00th=[12096], 90.00th=[12224], 95.00th=[12352],
     | 99.00th=[22144], 99.50th=[26752], 99.90th=[31616], 99.95th=[49408],
     | 99.99th=[63744]
   bw (  KiB/s): min=254784, max=257032, per=100.00%, avg=255632.36, stdev=623.78, samples=14
   iops        : min=63696, max=64258, avg=63908.00, stdev=155.97, samples=14
  lat (nsec)   : 1000=0.01%
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.55%, 50=1.40%
  lat (usec)   : 100=0.04%, 250=0.01%
  cpu          : usr=10.24%, sys=37.27%, ctx=447233, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=447249,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=250MiB/s (262MB/s), 250MiB/s-250MiB/s (262MB/s-262MB/s), io=1747MiB (1832MB), run=7001-7001msec

Disk stats (read/write):
  nvme0n1: ios=640196/0, merge=0/0, ticks=6020/0, in_queue=6020, util=99.09%

$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_0.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733344,
  "timestamp_ms" : 1666733344307,
  "time" : "Tue Oct 25 23:29:04 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2908217344,
        "io_kbytes" : 2840056,
        "bw_bytes" : 415400277,
        "bw" : 405664,
        "iops" : 101416.083417,
        "runtime" : 7001,
        "total_ios" : 710014,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1562,
          "max" : 34896,
          "mean" : 1634.305671,
          "stddev" : 120.389758,
          "N" : 710014
        },
        "clat_ns" : {
          "min" : 221,
          "max" : 113843,
          "mean" : 7995.203314,
          "stddev" : 1932.985662,
          "N" : 710014,
          "percentile" : {
            "1.000000" : 7520,
            "5.000000" : 7520,
            "10.000000" : 7520,
            "20.000000" : 7584,
            "30.000000" : 7584,
            "40.000000" : 7648,
            "50.000000" : 7712,
            "60.000000" : 7712,
            "70.000000" : 7776,
            "80.000000" : 7840,
            "90.000000" : 7840,
            "95.000000" : 8384,
            "99.000000" : 17280,
            "99.500000" : 19840,
            "99.900000" : 23680,
            "99.950000" : 40704,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 9027,
          "max" : 115476,
          "mean" : 9629.508986,
          "stddev" : 1936.932387,
          "N" : 710014
        },
        "bw_min" : 404904,
        "bw_max" : 407032,
        "bw_agg" : 100.000000,
        "bw_mean" : 405763.785714,
        "bw_dev" : 726.004893,
        "bw_samples" : 14,
        "iops_min" : 101226,
        "iops_max" : 101758,
        "iops_mean" : 101440.928571,
        "iops_stddev" : 181.477044,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 85.300000,
      "sys_cpu" : 14.671429,
      "ctx" : 18,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.102471,
        "20" : 3.421904,
        "50" : 0.436611,
        "100" : 0.037605,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85432: Tue Oct 25 23:29:04 2022
  read: IOPS=101k, BW=396MiB/s (415MB/s)(2773MiB/7001msec)
    slat (nsec): min=1562, max=34896, avg=1634.31, stdev=120.39
    clat (nsec): min=221, max=113843, avg=7995.20, stdev=1932.99
     lat (usec): min=9, max=115, avg= 9.63, stdev= 1.94
    clat percentiles (nsec):
     |  1.00th=[ 7520],  5.00th=[ 7520], 10.00th=[ 7520], 20.00th=[ 7584],
     | 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7712], 60.00th=[ 7712],
     | 70.00th=[ 7776], 80.00th=[ 7840], 90.00th=[ 7840], 95.00th=[ 8384],
     | 99.00th=[17280], 99.50th=[19840], 99.90th=[23680], 99.95th=[40704],
     | 99.99th=[65280]
   bw (  KiB/s): min=404904, max=407032, per=100.00%, avg=405763.79, stdev=726.00, samples=14
   iops        : min=101226, max=101758, avg=101440.93, stdev=181.48, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=96.10%, 20=3.42%, 50=0.44%, 100=0.04%, 250=0.01%
  cpu          : usr=85.30%, sys=14.67%, ctx=18, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=710014,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=396MiB/s (415MB/s), 396MiB/s-396MiB/s (415MB/s-415MB/s), io=2773MiB (2908MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_1.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733359,
  "timestamp_ms" : 1666733359618,
  "time" : "Tue Oct 25 23:29:19 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2864140288,
        "io_kbytes" : 2797012,
        "bw_bytes" : 409104454,
        "bw" : 399516,
        "iops" : 99879.017283,
        "runtime" : 7001,
        "total_ios" : 699253,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1643,
          "max" : 27060,
          "mean" : 1697.425543,
          "stddev" : 169.441086,
          "N" : 699253
        },
        "clat_ns" : {
          "min" : 210,
          "max" : 77946,
          "mean" : 8068.355484,
          "stddev" : 1872.173034,
          "N" : 699253,
          "percentile" : {
            "1.000000" : 7648,
            "5.000000" : 7712,
            "10.000000" : 7712,
            "20.000000" : 7712,
            "30.000000" : 7712,
            "40.000000" : 7712,
            "50.000000" : 7776,
            "60.000000" : 7776,
            "70.000000" : 7776,
            "80.000000" : 7776,
            "90.000000" : 7904,
            "95.000000" : 8512,
            "99.000000" : 17280,
            "99.500000" : 19328,
            "99.900000" : 23936,
            "99.950000" : 36608,
            "99.990000" : 64768
          }
        },
        "lat_ns" : {
          "min" : 9247,
          "max" : 79629,
          "mean" : 9765.781026,
          "stddev" : 1879.123838,
          "N" : 699253
        },
        "bw_min" : 398744,
        "bw_max" : 400280,
        "bw_agg" : 100.000000,
        "bw_mean" : 399610.785714,
        "bw_dev" : 354.371856,
        "bw_samples" : 14,
        "iops_min" : 99686,
        "iops_max" : 100070,
        "iops_mean" : 99902.642857,
        "iops_stddev" : 88.578177,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 85.242857,
      "sys_cpu" : 14.728571,
      "ctx" : 12,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.071522,
        "20" : 3.639741,
        "50" : 0.253127,
        "100" : 0.032034,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85481: Tue Oct 25 23:29:19 2022
  read: IOPS=99.9k, BW=390MiB/s (409MB/s)(2731MiB/7001msec)
    slat (nsec): min=1643, max=27060, avg=1697.43, stdev=169.44
    clat (nsec): min=210, max=77946, avg=8068.36, stdev=1872.17
     lat (nsec): min=9247, max=79629, avg=9765.78, stdev=1879.12
    clat percentiles (nsec):
     |  1.00th=[ 7648],  5.00th=[ 7712], 10.00th=[ 7712], 20.00th=[ 7712],
     | 30.00th=[ 7712], 40.00th=[ 7712], 50.00th=[ 7776], 60.00th=[ 7776],
     | 70.00th=[ 7776], 80.00th=[ 7776], 90.00th=[ 7904], 95.00th=[ 8512],
     | 99.00th=[17280], 99.50th=[19328], 99.90th=[23936], 99.95th=[36608],
     | 99.99th=[64768]
   bw (  KiB/s): min=398744, max=400280, per=100.00%, avg=399610.79, stdev=354.37, samples=14
   iops        : min=99686, max=100070, avg=99902.64, stdev=88.58, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 10=96.07%, 20=3.64%, 50=0.25%, 100=0.03%
  cpu          : usr=85.24%, sys=14.73%, ctx=12, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=699253,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=390MiB/s (409MB/s), 390MiB/s-390MiB/s (409MB/s-409MB/s), io=2731MiB (2864MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-bdev_aio_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_aio" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_aio_foo.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_aio_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_aio_2.txt
libaio-bdev_aio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733374,
  "timestamp_ms" : 1666733374688,
  "time" : "Tue Oct 25 23:29:34 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_aio",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_aio",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_aio_foo.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2917703680,
        "io_kbytes" : 2849320,
        "bw_bytes" : 416814811,
        "bw" : 407045,
        "iops" : 101761.428571,
        "runtime" : 7000,
        "total_ios" : 712330,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1533,
          "max" : 13756,
          "mean" : 1593.976842,
          "stddev" : 115.555755,
          "N" : 712330
        },
        "clat_ns" : {
          "min" : 221,
          "max" : 75782,
          "mean" : 7996.381760,
          "stddev" : 1896.986393,
          "N" : 712330,
          "percentile" : {
            "1.000000" : 7520,
            "5.000000" : 7520,
            "10.000000" : 7520,
            "20.000000" : 7584,
            "30.000000" : 7584,
            "40.000000" : 7648,
            "50.000000" : 7712,
            "60.000000" : 7712,
            "70.000000" : 7776,
            "80.000000" : 7840,
            "90.000000" : 7840,
            "95.000000" : 8256,
            "99.000000" : 17280,
            "99.500000" : 20096,
            "99.900000" : 22656,
            "99.950000" : 32384,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 9007,
          "max" : 77365,
          "mean" : 9590.358602,
          "stddev" : 1900.929895,
          "N" : 712330
        },
        "bw_min" : 406344,
        "bw_max" : 407800,
        "bw_agg" : 100.000000,
        "bw_mean" : 407090.071429,
        "bw_dev" : 420.417094,
        "bw_samples" : 14,
        "iops_min" : 101586,
        "iops_max" : 101950,
        "iops_mean" : 101772.500000,
        "iops_stddev" : 105.118797,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 85.871429,
      "sys_cpu" : 14.100000,
      "ctx" : 10,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.124268,
        "20" : 3.367821,
        "50" : 0.472955,
        "100" : 0.033692,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_aio: (groupid=0, jobs=1): err= 0: pid=85531: Tue Oct 25 23:29:34 2022
  read: IOPS=102k, BW=398MiB/s (417MB/s)(2783MiB/7000msec)
    slat (nsec): min=1533, max=13756, avg=1593.98, stdev=115.56
    clat (nsec): min=221, max=75782, avg=7996.38, stdev=1896.99
     lat (nsec): min=9007, max=77365, avg=9590.36, stdev=1900.93
    clat percentiles (nsec):
     |  1.00th=[ 7520],  5.00th=[ 7520], 10.00th=[ 7520], 20.00th=[ 7584],
     | 30.00th=[ 7584], 40.00th=[ 7648], 50.00th=[ 7712], 60.00th=[ 7712],
     | 70.00th=[ 7776], 80.00th=[ 7840], 90.00th=[ 7840], 95.00th=[ 8256],
     | 99.00th=[17280], 99.50th=[20096], 99.90th=[22656], 99.95th=[32384],
     | 99.99th=[65280]
   bw (  KiB/s): min=406344, max=407800, per=100.00%, avg=407090.07, stdev=420.42, samples=14
   iops        : min=101586, max=101950, avg=101772.50, stdev=105.12, samples=14
  lat (nsec)   : 250=0.01%, 500=0.01%
  lat (usec)   : 4=0.01%, 10=96.12%, 20=3.37%, 50=0.47%, 100=0.03%
  cpu          : usr=85.87%, sys=14.10%, ctx=10, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=712330,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=398MiB/s (417MB/s), 398MiB/s-398MiB/s (417MB/s-417MB/s), io=2783MiB (2918MB), run=7000-7000msec

$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_0.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733389,
  "timestamp_ms" : 1666733389939,
  "time" : "Tue Oct 25 23:29:49 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2542465024,
        "io_kbytes" : 2482876,
        "bw_bytes" : 363157409,
        "bw" : 354645,
        "iops" : 88661.476932,
        "runtime" : 7001,
        "total_ios" : 620719,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1783,
          "max" : 14928,
          "mean" : 1926.784324,
          "stddev" : 147.911807,
          "N" : 620719
        },
        "clat_ns" : {
          "min" : 611,
          "max" : 95610,
          "mean" : 9103.494547,
          "stddev" : 2144.724722,
          "N" : 620719,
          "percentile" : {
            "1.000000" : 8512,
            "5.000000" : 8640,
            "10.000000" : 8640,
            "20.000000" : 8640,
            "30.000000" : 8768,
            "40.000000" : 8768,
            "50.000000" : 8768,
            "60.000000" : 8768,
            "70.000000" : 8768,
            "80.000000" : 8896,
            "90.000000" : 9024,
            "95.000000" : 9536,
            "99.000000" : 19328,
            "99.500000" : 24960,
            "99.900000" : 26240,
            "99.950000" : 41728,
            "99.990000" : 65280
          }
        },
        "lat_ns" : {
          "min" : 9959,
          "max" : 97583,
          "mean" : 11030.278872,
          "stddev" : 2148.473123,
          "N" : 620719
        },
        "bw_min" : 353864,
        "bw_max" : 355863,
        "bw_agg" : 100.000000,
        "bw_mean" : 354830.928571,
        "bw_dev" : 468.121198,
        "bw_samples" : 14,
        "iops_min" : 88466,
        "iops_max" : 88965,
        "iops_mean" : 88707.642857,
        "iops_stddev" : 116.917201,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 34.157143,
      "sys_cpu" : 65.814286,
      "ctx" : 19,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 95.638445,
        "20" : 3.521722,
        "50" : 0.793274,
        "100" : 0.039309,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85581: Tue Oct 25 23:29:49 2022
  read: IOPS=88.7k, BW=346MiB/s (363MB/s)(2425MiB/7001msec)
    slat (nsec): min=1783, max=14928, avg=1926.78, stdev=147.91
    clat (nsec): min=611, max=95610, avg=9103.49, stdev=2144.72
     lat (nsec): min=9959, max=97583, avg=11030.28, stdev=2148.47
    clat percentiles (nsec):
     |  1.00th=[ 8512],  5.00th=[ 8640], 10.00th=[ 8640], 20.00th=[ 8640],
     | 30.00th=[ 8768], 40.00th=[ 8768], 50.00th=[ 8768], 60.00th=[ 8768],
     | 70.00th=[ 8768], 80.00th=[ 8896], 90.00th=[ 9024], 95.00th=[ 9536],
     | 99.00th=[19328], 99.50th=[24960], 99.90th=[26240], 99.95th=[41728],
     | 99.99th=[65280]
   bw (  KiB/s): min=353864, max=355863, per=100.00%, avg=354830.93, stdev=468.12, samples=14
   iops        : min=88466, max=88965, avg=88707.64, stdev=116.92, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 4=0.01%, 10=95.64%, 20=3.52%, 50=0.79%, 100=0.04%
  cpu          : usr=34.16%, sys=65.81%, ctx=19, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=620719,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=346MiB/s (363MB/s), 346MiB/s-346MiB/s (363MB/s-363MB/s), io=2425MiB (2542MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_1.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733404,
  "timestamp_ms" : 1666733404980,
  "time" : "Tue Oct 25 23:30:04 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2823823360,
        "io_kbytes" : 2757640,
        "bw_bytes" : 403403337,
        "bw" : 393948,
        "iops" : 98487.142857,
        "runtime" : 7000,
        "total_ios" : 689410,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1582,
          "max" : 36869,
          "mean" : 1652.357284,
          "stddev" : 116.694315,
          "N" : 689410
        },
        "clat_ns" : {
          "min" : 651,
          "max" : 112781,
          "mean" : 8251.688869,
          "stddev" : 1885.218026,
          "N" : 689410,
          "percentile" : {
            "1.000000" : 7776,
            "5.000000" : 7776,
            "10.000000" : 7776,
            "20.000000" : 7840,
            "30.000000" : 7840,
            "40.000000" : 7904,
            "50.000000" : 7904,
            "60.000000" : 7904,
            "70.000000" : 8032,
            "80.000000" : 8160,
            "90.000000" : 8256,
            "95.000000" : 8640,
            "99.000000" : 17280,
            "99.500000" : 19328,
            "99.900000" : 22656,
            "99.950000" : 41216,
            "99.990000" : 67072
          }
        },
        "lat_ns" : {
          "min" : 9198,
          "max" : 114405,
          "mean" : 9904.046153,
          "stddev" : 1889.202305,
          "N" : 689410
        },
        "bw_min" : 393288,
        "bw_max" : 394592,
        "bw_agg" : 100.000000,
        "bw_mean" : 393990.571429,
        "bw_dev" : 426.285876,
        "bw_samples" : 14,
        "iops_min" : 98322,
        "iops_max" : 98648,
        "iops_mean" : 98497.642857,
        "iops_stddev" : 106.571469,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 33.642857,
      "sys_cpu" : 66.328571,
      "ctx" : 17,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.010000,
        "10" : 96.026167,
        "20" : 3.730146,
        "50" : 0.206408,
        "100" : 0.035683,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85630: Tue Oct 25 23:30:04 2022
  read: IOPS=98.5k, BW=385MiB/s (403MB/s)(2693MiB/7000msec)
    slat (nsec): min=1582, max=36869, avg=1652.36, stdev=116.69
    clat (nsec): min=651, max=112781, avg=8251.69, stdev=1885.22
     lat (usec): min=9, max=114, avg= 9.90, stdev= 1.89
    clat percentiles (nsec):
     |  1.00th=[ 7776],  5.00th=[ 7776], 10.00th=[ 7776], 20.00th=[ 7840],
     | 30.00th=[ 7840], 40.00th=[ 7904], 50.00th=[ 7904], 60.00th=[ 7904],
     | 70.00th=[ 8032], 80.00th=[ 8160], 90.00th=[ 8256], 95.00th=[ 8640],
     | 99.00th=[17280], 99.50th=[19328], 99.90th=[22656], 99.95th=[41216],
     | 99.99th=[67072]
   bw (  KiB/s): min=393288, max=394592, per=100.00%, avg=393990.57, stdev=426.29, samples=14
   iops        : min=98322, max=98648, avg=98497.64, stdev=106.57, samples=14
  lat (nsec)   : 750=0.01%
  lat (usec)   : 4=0.01%, 10=96.03%, 20=3.73%, 50=0.21%, 100=0.04%
  lat (usec)   : 250=0.01%
  cpu          : usr=33.64%, sys=66.33%, ctx=17, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=689410,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=385MiB/s (403MB/s), 385MiB/s-385MiB/s (403MB/s-403MB/s), io=2693MiB (2824MB), run=7000-7000msec

$ rm /tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-bdev_xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="spdk_bdev" --spdk_conf="/tmp/spdk_bdev_xnvme_libaio.conf" --filename="Nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt"
TELEMETRY: No legacy callbacks, legacy socket not created

$ cat /tmp/fio-output_4K_1_libaio-bdev_xnvme_2.txt
libaio-bdev_xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk_bdev, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733420,
  "timestamp_ms" : 1666733420440,
  "time" : "Tue Oct 25 23:30:20 2022",
  "jobs" : [
    {
      "jobname" : "libaio-bdev_xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-bdev_xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "spdk_bdev",
        "spdk_conf" : "/tmp/spdk_bdev_xnvme_libaio.conf",
        "filename" : "Nvme0n1"
      },
      "read" : {
        "io_bytes" : 2837221376,
        "io_kbytes" : 2770724,
        "bw_bytes" : 405259445,
        "bw" : 395761,
        "iops" : 98940.294244,
        "runtime" : 7001,
        "total_ios" : 692681,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 1563,
          "max" : 52007,
          "mean" : 1627.296386,
          "stddev" : 124.623637,
          "N" : 692681
        },
        "clat_ns" : {
          "min" : 631,
          "max" : 93886,
          "mean" : 8236.372760,
          "stddev" : 1870.436580,
          "N" : 692681,
          "percentile" : {
            "1.000000" : 7584,
            "5.000000" : 7648,
            "10.000000" : 7840,
            "20.000000" : 7904,
            "30.000000" : 7904,
            "40.000000" : 7904,
            "50.000000" : 7968,
            "60.000000" : 7968,
            "70.000000" : 7968,
            "80.000000" : 7968,
            "90.000000" : 8032,
            "95.000000" : 8640,
            "99.000000" : 17280,
            "99.500000" : 19328,
            "99.900000" : 22656,
            "99.950000" : 40704,
            "99.990000" : 66048
          }
        },
        "lat_ns" : {
          "min" : 9167,
          "max" : 95499,
          "mean" : 9863.669146,
          "stddev" : 1874.856856,
          "N" : 692681
        },
        "bw_min" : 395336,
        "bw_max" : 396672,
        "bw_agg" : 100.000000,
        "bw_mean" : 395852.500000,
        "bw_dev" : 386.086230,
        "bw_samples" : 14,
        "iops_min" : 98834,
        "iops_max" : 99168,
        "iops_mean" : 98963.071429,
        "iops_stddev" : 96.554858,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 28.842857,
      "sys_cpu" : 71.128571,
      "ctx" : 44,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.010000,
        "1000" : 0.010000
      },
      "latency_us" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 96.036992,
        "20" : 3.718595,
        "50" : 0.209476,
        "100" : 0.033926,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-bdev_xnvme: (groupid=0, jobs=1): err= 0: pid=85679: Tue Oct 25 23:30:20 2022
  read: IOPS=98.9k, BW=386MiB/s (405MB/s)(2706MiB/7001msec)
    slat (nsec): min=1563, max=52007, avg=1627.30, stdev=124.62
    clat (nsec): min=631, max=93886, avg=8236.37, stdev=1870.44
     lat (nsec): min=9167, max=95499, avg=9863.67, stdev=1874.86
    clat percentiles (nsec):
     |  1.00th=[ 7584],  5.00th=[ 7648], 10.00th=[ 7840], 20.00th=[ 7904],
     | 30.00th=[ 7904], 40.00th=[ 7904], 50.00th=[ 7968], 60.00th=[ 7968],
     | 70.00th=[ 7968], 80.00th=[ 7968], 90.00th=[ 8032], 95.00th=[ 8640],
     | 99.00th=[17280], 99.50th=[19328], 99.90th=[22656], 99.95th=[40704],
     | 99.99th=[66048]
   bw (  KiB/s): min=395336, max=396672, per=100.00%, avg=395852.50, stdev=386.09, samples=14
   iops        : min=98834, max=99168, avg=98963.07, stdev=96.55, samples=14
  lat (nsec)   : 750=0.01%, 1000=0.01%
  lat (usec)   : 10=96.04%, 20=3.72%, 50=0.21%, 100=0.03%
  cpu          : usr=28.84%, sys=71.13%, ctx=44, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=692681,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=386MiB/s (405MB/s), 386MiB/s-386MiB/s (405MB/s-405MB/s), io=2706MiB (2837MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-xnvme_0.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_0.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_0.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_0.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733434,
  "timestamp_ms" : 1666733434249,
  "time" : "Tue Oct 25 23:30:34 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1678741504,
        "io_kbytes" : 1639396,
        "bw_bytes" : 239785959,
        "bw" : 234165,
        "iops" : 58541.494072,
        "runtime" : 7001,
        "total_ios" : 409849,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2966,
          "max" : 24035,
          "mean" : 3340.552930,
          "stddev" : 206.631663,
          "N" : 409849
        },
        "clat_ns" : {
          "min" : 1123,
          "max" : 438051,
          "mean" : 13173.901883,
          "stddev" : 2319.222617,
          "N" : 409849,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 12096,
            "10.000000" : 12096,
            "20.000000" : 12224,
            "30.000000" : 12352,
            "40.000000" : 12480,
            "50.000000" : 13376,
            "60.000000" : 13376,
            "70.000000" : 13376,
            "80.000000" : 13504,
            "90.000000" : 13504,
            "95.000000" : 13632,
            "99.000000" : 23680,
            "99.500000" : 28800,
            "99.900000" : 35072,
            "99.950000" : 50944,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 12203,
          "max" : 441297,
          "mean" : 16514.454814,
          "stddev" : 2343.548276,
          "N" : 409849
        },
        "bw_min" : 226544,
        "bw_max" : 245707,
        "bw_agg" : 100.000000,
        "bw_mean" : 234225.357143,
        "bw_dev" : 8659.960984,
        "bw_samples" : 14,
        "iops_min" : 56636,
        "iops_max" : 61426,
        "iops_mean" : 58556.285714,
        "iops_stddev" : 2164.913764,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.657143,
      "sys_cpu" : 36.671429,
      "ctx" : 409826,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.453089,
        "50" : 1.489085,
        "100" : 0.050506,
        "250" : 0.010000,
        "500" : 0.010000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85718: Tue Oct 25 23:30:34 2022
  read: IOPS=58.5k, BW=229MiB/s (240MB/s)(1601MiB/7001msec)
    slat (nsec): min=2966, max=24035, avg=3340.55, stdev=206.63
    clat (nsec): min=1123, max=438051, avg=13173.90, stdev=2319.22
     lat (usec): min=12, max=441, avg=16.51, stdev= 2.34
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
     | 30.00th=[12352], 40.00th=[12480], 50.00th=[13376], 60.00th=[13376],
     | 70.00th=[13376], 80.00th=[13504], 90.00th=[13504], 95.00th=[13632],
     | 99.00th=[23680], 99.50th=[28800], 99.90th=[35072], 99.95th=[50944],
     | 99.99th=[73216]
   bw (  KiB/s): min=226544, max=245707, per=100.00%, avg=234225.36, stdev=8659.96, samples=14
   iops        : min=56636, max=61426, avg=58556.29, stdev=2164.91, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.45%, 50=1.49%
  lat (usec)   : 100=0.05%, 250=0.01%, 500=0.01%
  cpu          : usr=12.66%, sys=36.67%, ctx=409826, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=409849,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=229MiB/s (240MB/s), 229MiB/s-229MiB/s (240MB/s-240MB/s), io=1601MiB (1679MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-xnvme_1.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_1.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_1.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_1.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733448,
  "timestamp_ms" : 1666733448033,
  "time" : "Tue Oct 25 23:30:48 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1764671488,
        "io_kbytes" : 1723312,
        "bw_bytes" : 252059918,
        "bw" : 246152,
        "iops" : 61538.065991,
        "runtime" : 7001,
        "total_ios" : 430828,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2925,
          "max" : 175278,
          "mean" : 3159.793955,
          "stddev" : 387.428894,
          "N" : 430828
        },
        "clat_ns" : {
          "min" : 1002,
          "max" : 182192,
          "mean" : 12504.355956,
          "stddev" : 2223.696600,
          "N" : 430828,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 11968,
            "10.000000" : 12096,
            "20.000000" : 12096,
            "30.000000" : 12096,
            "40.000000" : 12224,
            "50.000000" : 12224,
            "60.000000" : 12224,
            "70.000000" : 12352,
            "80.000000" : 12352,
            "90.000000" : 12480,
            "95.000000" : 12608,
            "99.000000" : 22144,
            "99.500000" : 29056,
            "99.900000" : 37632,
            "99.950000" : 52992,
            "99.990000" : 73216
          }
        },
        "lat_ns" : {
          "min" : 12213,
          "max" : 188343,
          "mean" : 15664.149911,
          "stddev" : 2259.446273,
          "N" : 430828
        },
        "bw_min" : 245008,
        "bw_max" : 247408,
        "bw_agg" : 100.000000,
        "bw_mean" : 246227.357143,
        "bw_dev" : 707.156343,
        "bw_samples" : 14,
        "iops_min" : 61252,
        "iops_max" : 61852,
        "iops_mean" : 61556.785714,
        "iops_stddev" : 176.840727,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 13.242857,
      "sys_cpu" : 39.485714,
      "ctx" : 430779,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.000000,
        "10" : 0.010213,
        "20" : 98.469227,
        "50" : 1.455801,
        "100" : 0.060581,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85758: Tue Oct 25 23:30:48 2022
  read: IOPS=61.5k, BW=240MiB/s (252MB/s)(1683MiB/7001msec)
    slat (nsec): min=2925, max=175278, avg=3159.79, stdev=387.43
    clat (nsec): min=1002, max=182192, avg=12504.36, stdev=2223.70
     lat (usec): min=12, max=188, avg=15.66, stdev= 2.26
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[11968], 10.00th=[12096], 20.00th=[12096],
     | 30.00th=[12096], 40.00th=[12224], 50.00th=[12224], 60.00th=[12224],
     | 70.00th=[12352], 80.00th=[12352], 90.00th=[12480], 95.00th=[12608],
     | 99.00th=[22144], 99.50th=[29056], 99.90th=[37632], 99.95th=[52992],
     | 99.99th=[73216]
   bw (  KiB/s): min=245008, max=247408, per=100.00%, avg=246227.36, stdev=707.16, samples=14
   iops        : min=61252, max=61852, avg=61556.79, stdev=176.84, samples=14
  lat (usec)   : 2=0.01%, 10=0.01%, 20=98.47%, 50=1.46%, 100=0.06%
  lat (usec)   : 250=0.01%
  cpu          : usr=13.24%, sys=39.49%, ctx=430779, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=430828,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=240MiB/s (252MB/s), 240MiB/s-240MiB/s (252MB/s-252MB/s), io=1683MiB (1765MB), run=7001-7001msec

$ rm /tmp/fio-output_4K_1_libaio-xnvme_2.txt
rm: cannot remove '/tmp/fio-output_4K_1_libaio-xnvme_2.txt': No such file or directory

$ /opt/fio/bin/fio --name="libaio-xnvme" --rw="randread" --size="12G" --iodepth="1" --bs="4K" --direct="1" --thread="1" --time_based="1" --runtime="7" --ramp_time="3" --norandommap="1" --allow_file_create="0" --ioengine="external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so" --xnvme_be="linux" --xnvme_async="libaio" --xnvme_sync="nvme" --xnvme_admin="nvme" --xnvme_dev_nsid="1" --filename="/dev/nvme0n1" --output-format="normal,json" --output="/tmp/fio-output_4K_1_libaio-xnvme_2.txt"
$ cat /tmp/fio-output_4K_1_libaio-xnvme_2.txt
libaio-xnvme: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=xnvme, iodepth=1
fio-3.32-dirty
Starting 1 thread
{
  "fio version" : "fio-3.32-dirty",
  "timestamp" : 1666733461,
  "timestamp_ms" : 1666733461706,
  "time" : "Tue Oct 25 23:31:01 2022",
  "jobs" : [
    {
      "jobname" : "libaio-xnvme",
      "groupid" : 0,
      "error" : 0,
      "eta" : 0,
      "elapsed" : 11,
      "job options" : {
        "name" : "libaio-xnvme",
        "rw" : "randread",
        "size" : "12G",
        "iodepth" : "1",
        "bs" : "4K",
        "direct" : "1",
        "thread" : "1",
        "time_based" : "1",
        "runtime" : "7",
        "ramp_time" : "3",
        "norandommap" : "1",
        "allow_file_create" : "0",
        "ioengine" : "external:/usr/local/lib/x86_64-linux-gnu/libxnvme-fio-engine.so",
        "xnvme_be" : "linux",
        "xnvme_async" : "libaio",
        "xnvme_sync" : "nvme",
        "xnvme_admin" : "nvme",
        "xnvme_dev_nsid" : "1",
        "filename" : "/dev/nvme0n1"
      },
      "read" : {
        "io_bytes" : 1754910720,
        "io_kbytes" : 1713780,
        "bw_bytes" : 250665722,
        "bw" : 244790,
        "iops" : 61197.686045,
        "runtime" : 7001,
        "total_ios" : 428445,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 2905,
          "max" : 25257,
          "mean" : 3138.958107,
          "stddev" : 264.649411,
          "N" : 428445
        },
        "clat_ns" : {
          "min" : 1072,
          "max" : 196789,
          "mean" : 12600.985377,
          "stddev" : 2183.473992,
          "N" : 428445,
          "percentile" : {
            "1.000000" : 11968,
            "5.000000" : 12096,
            "10.000000" : 12096,
            "20.000000" : 12224,
            "30.000000" : 12224,
            "40.000000" : 12224,
            "50.000000" : 12352,
            "60.000000" : 12352,
            "70.000000" : 12480,
            "80.000000" : 12480,
            "90.000000" : 12480,
            "95.000000" : 12736,
            "99.000000" : 22400,
            "99.500000" : 29312,
            "99.900000" : 32384,
            "99.950000" : 50944,
            "99.990000" : 72192
          }
        },
        "lat_ns" : {
          "min" : 12002,
          "max" : 199895,
          "mean" : 15739.943484,
          "stddev" : 2203.074385,
          "N" : 428445
        },
        "bw_min" : 243672,
        "bw_max" : 246248,
        "bw_agg" : 100.000000,
        "bw_mean" : 244847.571429,
        "bw_dev" : 682.736655,
        "bw_samples" : 14,
        "iops_min" : 60918,
        "iops_max" : 61562,
        "iops_mean" : 61211.857143,
        "iops_stddev" : 170.666502,
        "iops_samples" : 14
      },
      "write" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "trim" : {
        "io_bytes" : 0,
        "io_kbytes" : 0,
        "bw_bytes" : 0,
        "bw" : 0,
        "iops" : 0.000000,
        "runtime" : 0,
        "total_ios" : 0,
        "short_ios" : 0,
        "drop_ios" : 0,
        "slat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "clat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        },
        "bw_min" : 0,
        "bw_max" : 0,
        "bw_agg" : 0.000000,
        "bw_mean" : 0.000000,
        "bw_dev" : 0.000000,
        "bw_samples" : 0,
        "iops_min" : 0,
        "iops_max" : 0,
        "iops_mean" : 0.000000,
        "iops_stddev" : 0.000000,
        "iops_samples" : 0
      },
      "sync" : {
        "total_ios" : 0,
        "lat_ns" : {
          "min" : 0,
          "max" : 0,
          "mean" : 0.000000,
          "stddev" : 0.000000,
          "N" : 0
        }
      },
      "job_runtime" : 7000,
      "usr_cpu" : 12.700000,
      "sys_cpu" : 39.757143,
      "ctx" : 428394,
      "majf" : 0,
      "minf" : 0,
      "iodepth_level" : {
        "1" : 100.000000,
        "2" : 0.000000,
        "4" : 0.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_submit" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "iodepth_complete" : {
        "0" : 0.000000,
        "4" : 100.000000,
        "8" : 0.000000,
        "16" : 0.000000,
        "32" : 0.000000,
        "64" : 0.000000,
        ">=64" : 0.000000
      },
      "latency_ns" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_us" : {
        "2" : 0.010000,
        "4" : 0.010000,
        "10" : 0.010000,
        "20" : 98.445308,
        "50" : 1.491907,
        "100" : 0.050415,
        "250" : 0.010000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000
      },
      "latency_ms" : {
        "2" : 0.000000,
        "4" : 0.000000,
        "10" : 0.000000,
        "20" : 0.000000,
        "50" : 0.000000,
        "100" : 0.000000,
        "250" : 0.000000,
        "500" : 0.000000,
        "750" : 0.000000,
        "1000" : 0.000000,
        "2000" : 0.000000,
        ">=2000" : 0.000000
      },
      "latency_depth" : 1,
      "latency_target" : 0,
      "latency_percentile" : 100.000000,
      "latency_window" : 0
    }
  ]
}

libaio-xnvme: (groupid=0, jobs=1): err= 0: pid=85798: Tue Oct 25 23:31:01 2022
  read: IOPS=61.2k, BW=239MiB/s (251MB/s)(1674MiB/7001msec)
    slat (nsec): min=2905, max=25257, avg=3138.96, stdev=264.65
    clat (nsec): min=1072, max=196789, avg=12600.99, stdev=2183.47
     lat (usec): min=12, max=199, avg=15.74, stdev= 2.20
    clat percentiles (nsec):
     |  1.00th=[11968],  5.00th=[12096], 10.00th=[12096], 20.00th=[12224],
     | 30.00th=[12224], 40.00th=[12224], 50.00th=[12352], 60.00th=[12352],
     | 70.00th=[12480], 80.00th=[12480], 90.00th=[12480], 95.00th=[12736],
     | 99.00th=[22400], 99.50th=[29312], 99.90th=[32384], 99.95th=[50944],
     | 99.99th=[72192]
   bw (  KiB/s): min=243672, max=246248, per=100.00%, avg=244847.57, stdev=682.74, samples=14
   iops        : min=60918, max=61562, avg=61211.86, stdev=170.67, samples=14
  lat (usec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=98.45%, 50=1.49%
  lat (usec)   : 100=0.05%, 250=0.01%
  cpu          : usr=12.70%, sys=39.76%, ctx=428394, majf=0, minf=0
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=428445,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=239MiB/s (251MB/s), 239MiB/s-239MiB/s (251MB/s-251MB/s), io=1674MiB (1755MB), run=7001-7001msec

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Extract and transform fio and bdevperf output
    =============================================

    fio has:

    * bandwidth in KiB / second
    * latency is mean and in nano-seconds
    * IOPS are 'raw' / base-unit

    bdevperf has:

    * bandwidth in MiB / second
    * IOPS are 'raw' / base-unit

    NOTE: The metric-context need addition of backend-options, the options are
    semi-encoded by ioengine and 'label', however, that is not very precise.

{'description': '\n'
                '    Extract and transform fio and bdevperf output\n'
                '    =============================================\n'
                '\n'
                '    fio has:\n'
                '\n'
                '    * bandwidth in KiB / second\n'
                '    * latency is mean and in nano-seconds\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    bdevperf has:\n'
                '\n'
                '    * bandwidth in MiB / second\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    NOTE: The metric-context need addition of '
                'backend-options, the options are\n'
                "    semi-encoded by ioengine and 'label', however, that is "
                'not very precise.\n',
 'extras': {},
 'id': '003_plot_fio',
 'name': 'plot_fio',
 'nr': 3,
 'status': {'elapsed': 0.7708134651184082,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output', 'tool': 'fio'}}

Steps uses worklets. A description of the worklet used by the current step is provided below. The description is extracted from the docstring of the worklet and provided here in verbatim / as-is. 


    Extract and transform fio and bdevperf output
    =============================================

    fio has:

    * bandwidth in KiB / second
    * latency is mean and in nano-seconds
    * IOPS are 'raw' / base-unit

    bdevperf has:

    * bandwidth in MiB / second
    * IOPS are 'raw' / base-unit

    NOTE: The metric-context need addition of backend-options, the options are
    semi-encoded by ioengine and 'label', however, that is not very precise.

{'description': '\n'
                '    Extract and transform fio and bdevperf output\n'
                '    =============================================\n'
                '\n'
                '    fio has:\n'
                '\n'
                '    * bandwidth in KiB / second\n'
                '    * latency is mean and in nano-seconds\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    bdevperf has:\n'
                '\n'
                '    * bandwidth in MiB / second\n'
                "    * IOPS are 'raw' / base-unit\n"
                '\n'
                '    NOTE: The metric-context need addition of '
                'backend-options, the options are\n'
                "    semi-encoded by ioengine and 'label', however, that is "
                'not very precise.\n',
 'extras': {},
 'id': '004_plot_bdevperf',
 'name': 'plot_bdevperf',
 'nr': 4,
 'status': {'elapsed': 0.48505663871765137,
            'failed': 0,
            'passed': 1,
            'skipped': 0,
            'started': 0.0},
 'uses': 'plot',
 'with': {'path': 'cijoe-output', 'tool': 'bdevperf'}}
EOL: In the presence of failure, recall, this too shall pass.